date:20171016

[jira] [Comment Edited] (YARN-6620) Add support in NodeManager to isolate GPU devices by using CGroups

2017-10-16 Thread Zhankun Tang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207026#comment-16207026
 ] 

Zhankun Tang edited comment on YARN-6620 at 10/17/17 5:43 AM:
--

[~wangda], Thanks for the great effort. I'll implement the FPGA resource 
plugin(only supports OpenCL FPGA application for now) based on that.

One question is that in above discussions you mentioned the GPU is a mandatory 
resource like cpu and memory, but what's the difference between mandatory and 
first-class resource? And a list of first-class resource? Currently, If I 
understand correctly,
1. First-class resource should be parsed from resource-types.xml and 
node-resources.xml(or auto discover) instead of yarn-site.xml?
2. First-calss resource handler should register itself with the same resource 
name defined in xml files?
3. First-class resource should be shown in a separate user-defined column in 
web pages?


was (Author: tangzhankun):
[~wangda], Thanks for the great effort. I'll implement the FPGA resource 
plugin(only supports OpenCL FPGA application for now) based on that.

One question is that in above discussions you mentioned the GPU is a mandatory 
resource like cpu and memory, but what's the difference between mandatory and 
first-class resource? And a list of first-class resource? Currently, If I 
understand correctly,
1. First-class resource should be parsed from resource-types.xml and 
node-resources.xml instead of yarn-site.xml?
2. First-calss resource handler should register itself with the same resource 
name defined in xml files?
3. First-class resource should be shown in a separate user-defined column in 
web pages?

> Add support in NodeManager to isolate GPU devices by using CGroups
> --
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.1.0
>
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, 
> YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, 
> YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, 
> YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6620) Add support in NodeManager to isolate GPU devices by using CGroups

2017-10-16 Thread Zhankun Tang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207026#comment-16207026
 ] 

Zhankun Tang commented on YARN-6620:


[~wangda], Thanks for the great effort. I'll implement the FPGA resource 
plugin(only supports OpenCL FPGA application for now) based on that.

One question is that in above discussions you mentioned the GPU is a mandatory 
resource like cpu and memory, but what's the difference between mandatory and 
first-class resource? And a list of first-class resource? Currently, If I 
understand correctly,
1. First-class resource should be parsed from resource-types.xml and 
node-resources.xml instead of yarn-site.xml?
2. First-calss resource handler should register itself with the same resource 
name defined in xml files?
3. First-class resource should be shown in a separate user-defined column in 
web pages?

> Add support in NodeManager to isolate GPU devices by using CGroups
> --
>
> Key: YARN-6620
> URL: https://issues.apache.org/jira/browse/YARN-6620
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.1.0
>
> Attachments: YARN-6620.001.patch, YARN-6620.002.patch, 
> YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, 
> YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, 
> YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, 
> YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, 
> YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch
>
>
> This JIRA plan to add support of:
> 1) GPU configuration for NodeManagers
> 2) Isolation in CGroups. (Java side).
> 3) NM restart and recovery allocated GPU devices



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7275) NM Statestore cleanup for Container updates

2017-10-16 Thread kartheek muthyala (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16207007#comment-16207007
 ] 

kartheek muthyala commented on YARN-7275:
-

Thank you [~asuresh], for the review and the commit.

> NM Statestore cleanup for Container updates
> ---
>
> Key: YARN-7275
> URL: https://issues.apache.org/jira/browse/YARN-7275
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: kartheek muthyala
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7275.001.patch, YARN-7275.002.patch, 
> YARN-7275.003.patch, YARN-7275.004.patch, YARN-7275.005.patch, 
> YARN-7275.006.patch
>
>
> Currently, only resource updates are recorded in the NM state store, we need 
> to add ExecutionType updates as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7343) Add a junit test for ContainerScheduler recovery

2017-10-16 Thread kartheek muthyala (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

kartheek muthyala updated YARN-7343:

Target Version/s: 2.9.0, 3.0.0
   Fix Version/s: (was: 3.0.0)
  (was: 2.9.0)

> Add a junit test for ContainerScheduler recovery
> 
>
> Key: YARN-7343
> URL: https://issues.apache.org/jira/browse/YARN-7343
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: kartheek muthyala
>Assignee: kartheek muthyala
>Priority: Minor
>
> With queuing at NM, Container recovery becomes interesting. Add a junit test 
> for recovering containers in different states. This should test the recovery 
> with the ContainerScheduler class that was introduced for enabling container 
> queuing on contention of resources. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7343) Add a junit test for ContainerScheduler recovery

2017-10-16 Thread kartheek muthyala (JIRA)

kartheek muthyala created YARN-7343:
---

 Summary: Add a junit test for ContainerScheduler recovery
 Key: YARN-7343
 URL: https://issues.apache.org/jira/browse/YARN-7343
 Project: Hadoop YARN
  Issue Type: Task
Reporter: kartheek muthyala
Assignee: kartheek muthyala
Priority: Minor
 Fix For: 2.9.0, 3.0.0


With queuing at NM, Container recovery becomes interesting. Add a junit test 
for recovering containers in different states. This should test the recovery 
with the ContainerScheduler class that was introduced for enabling container 
queuing on contention of resources. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.

2017-10-16 Thread Sunil G (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206987#comment-16206987
 ] 

Sunil G commented on YARN-7338:
---

UI2 is launched as a separate service context in same RM webserver. And UI2 is 
available in {{:/ui2}} which serves just static pages. 

New YARN UI2 is a SPA (single page application) which downloads statics from  
{{:/ui2}}. And then onwards like any general REST client, browser 
which loaded UI2 statics will contact RM and NM via server's secure REST end 
points. Hence UI2 does not impose any issues as of today.

Hence I do not think we have to add these filters to UI2 or not. If its needed, 
we can add filters to ui2 context also, but i would like to hear thoughts from 
all folks. [~leftnoteasy] [~vrushalic] [~eyang] [~Sreenath] Please help to 
share your comments.

> Support same origin policy for cross site scripting prevention.
> ---
>
> Key: YARN-7338
> URL: https://issues.apache.org/jira/browse/YARN-7338
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Vrushali C
>
> Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new 
> web UI) to branch2  
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E
> --
> Ui2 does not seem to support same origin policy for cross site scripting 
> prevention.
> The following parameters has no effect for /ui2:
> hadoop.http.cross-origin.enabled = true
> yarn.resourcemanager.webapp.cross-origin.enabled = true
> This is because ui2 is designed as a separate web application.  WebFilters 
> setup for existing resource manager doesn’t apply to the new web application.
> Please open JIRA to track the security issue and resolve the problem prior to 
> backporting this to branch-2.
> This would minimize the risk to open up security hole in branch-2.
> --



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7339) LocalityMulticastAMRMProxyPolicy should handle cancel request properly

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206971#comment-16206971
 ] 

Hadoop QA commented on YARN-7339:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 29s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 29s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
58s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7339 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892514/YARN-7339-v2.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux e5289dd9af7e 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 31ebccc |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/17980/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17980/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> LocalityMulticastAMRMProxyPolicy should handle cancel request properly
> --
>
> Key: YARN-7339
> URL:

[jira] [Commented] (YARN-7013) merge related work for YARN-3926 branch

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206969#comment-16206969
 ] 

Hadoop QA commented on YARN-7013:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 46 new or modified test 
files. {color} |
|| || || || {color:brown} branch-3.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
58s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
25s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
17s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  6m 
51s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 28s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
13s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  4m 
40s{color} | {color:green} branch-3.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m 
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 42s{color} 
| {color:red} root generated 1 new + 1272 unchanged - 0 fixed = 1273 total (was 
1272) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 16s{color} | {color:orange} root: The patch generated 57 new + 2328 
unchanged - 67 fixed = 2385 total (was 2395) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  7m 
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
16s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 30s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  9m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
52s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 
5400 unchanged - 6 fixed = 5400 total (was 5406) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common 
generated 0 new + 4458 unchanged - 5 fixed = 4458 total (was 4463) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color}

[jira] [Commented] (YARN-7323) Data structure update in service REST API

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206966#comment-16206966
 ] 

Hadoop QA commented on YARN-7323:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
26s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 5s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
16s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
13s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 11 new + 165 unchanged - 9 fixed = 176 total (was 174) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 56s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
29s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
26s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
18s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
31s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7323 |
| JIRA

[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups

2017-10-16 Thread Wangda Tan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206962#comment-16206962
 ] 

Wangda Tan commented on YARN-6852:
--

[~jhung], thanks for reporting this. Yeah you're correct, in real world almost 
all minor_numbers of GPU start with 0 and ended with n-1, so that's why we 
didn't see this issue in test. Please file a ticket to track this issue. 

> [YARN-6223] Native code changes to support isolate GPU devices by using 
> CGroups
> ---
>
> Key: YARN-6852
> URL: https://issues.apache.org/jira/browse/YARN-6852
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6852.001.patch, YARN-6852.002.patch, 
> YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, 
> YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, 
> YARN-6852.009.patch
>
>
> This JIRA plan to add support of:
> 1) Isolation in CGroups. (native side).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7339) LocalityMulticastAMRMProxyPolicy should handle cancel request properly

2017-10-16 Thread Botong Huang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-7339:
---
Attachment: YARN-7339-v2.patch

Retry with same patch as v2

> LocalityMulticastAMRMProxyPolicy should handle cancel request properly
> --
>
> Key: YARN-7339
> URL: https://issues.apache.org/jira/browse/YARN-7339
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-7339-v1.patch, YARN-7339-v2.patch
>
>
> Currently inside AMRMProxy, LocalityMulticastAMRMProxyPolicy is not handling 
> and splitting cancel requests from AM properly: 
> # For node cancel request, we should not treat it as a localized resource 
> request. Otherwise it can lead to all weight zero issue when computing 
> localized resource weight. 
> # For ANY cancel, we should broadcast to all known subclusters, not just the 
> ones associated with localized resources. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6492) Generate queue metrics for each partition

2017-10-16 Thread Manikandan R (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206928#comment-16206928
 ] 

Manikandan R commented on YARN-6492:


[~jhung] Thanks for your comments.

{quote}What's the rationale behind having a separate PartitionQueueMetrics 
metrics info?{quote}

Currently as of now we have been following "Partition->Queue" hierarchy in UI 
and scheduling as well. To ensure that we have similar hierarchy even in JMX 
metrics and also helps users/admins to have a better understanding about the 
usage.

{quote}s it possible to factor the partition metrics to QueueMetrics class 
similar to how user metrics is implemented? (i.e. have some configuration to 
enable partition metrics, and register a new QueueMetrics with the partition 
tag){quote}

Yes, thats the plan. We will do this while implementing the same.

{quote}Also the existing metrics for PartitionQueueMetrics in the attachment 
only has pendingMB, pendingVCores, pendingContainers. Will this be filled out 
to match the set of metrics supported by the current queue metrics? 
(appsSubmitted, appsFailed, allocatedMB, availableMB, ...){quote}

Yes. Earlier attached o/p was to detail out the structure to help us in better 
understanding for further discussions purpose. While doing the implementation, 
we will cover all the existing metrics.

> Generate queue metrics for each partition
> -
>
> Key: YARN-6492
> URL: https://issues.apache.org/jira/browse/YARN-6492
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: capacity scheduler
>Reporter: Jonathan Hung
>Assignee: Manikandan R
> Attachments: YARN-6492.001.patch, partition_metrics.txt
>
>
> We are interested in having queue metrics for all partitions. Right now each 
> queue has one QueueMetrics object which captures metrics either in default 
> partition or across all partitions. (After YARN-6467 it will be in default 
> partition)
> But having the partition metrics would be very useful.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206912#comment-16206912
 ] 

Hadoop QA commented on YARN-7197:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
 7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
35m 16s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 58s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 59s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
29s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 57s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7197 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892502/YARN-7197.001.patch |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux a7fb3ca13e1b 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / b406d8e |
| Default Java | 1.8.0_144 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/17978/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/17978/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: 
hadoop-yarn-project/hadoop-yarn |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17978/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Add support for a volume blacklist for docker containers
> 
>
> Key: YARN-7197
> URL:

[jira] [Commented] (YARN-7323) Data structure update in service REST API

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206866#comment-16206866
 ] 

Hadoop QA commented on YARN-7323:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
44s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
55s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
13s{color} | {color:green} yarn-native-services passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
52s{color} | {color:red} hadoop-yarn-services-core in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 45s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
31s{color} | {color:red} hadoop-yarn-services-core in yarn-native-services 
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
23s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color} 
|
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
19s{color} | {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 11 new + 166 unchanged - 9 fixed = 177 total (was 175) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
35s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 21s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
35s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 31s{color} 
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
41s{color} | {color:green} The patch does not

[jira] [Comment Edited] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206825#comment-16206825
 ] 

Yufei Gu edited comment on YARN-4859 at 10/17/17 12:58 AM:
---

[~asuresh], thanks for the explanation. I tested either drf and fair policy. 
Both works. The trick is to set reservation-request vcores to zero for fair 
policy queues. This is due to the reservation calculation is based on steady 
fair share, whose vcores always be zero for fair policy queues. We could do 
some improvement, like ignoring the vcores if the request is for fair policy 
queues. Probably create another JIRA for this.
[~subru], I tested creating reservation and submitting a job with the 
reservation id, it works without YARN-7311. We still need to fix the 
"root.root" queue issue though. But "root.root" queue issue should be 
considered as a reservation bug, neither a blocker. Therefore, we can close 
this one. Besides that, YARN-7311 should be considered as a blocker. Could you 
spare time to review it? Thanks.


was (Author: yufeigu):
[~asuresh], thanks for the explanation. I tested either drf and fair policy. 
Both works. The trick is to set reservation-request vcores to zero for fair 
policy queues. This is due to the reservation calculation is based on steady 
fair share, whose vcores always be zero for fair policy queues. We could do 
some improvement, like ignoring the vcores if the request is for fair policy 
queues. Probably create another JIRA for this.
[~subru], I tested creating reservation and submitting a job with the 
reservation id, it works without YARN-7311. We still need to fix the 
"root.root" queue issue though. I'll post the patch later. Besides that, 
YARN-7311 should be considered as a blocker as well. Could you spare time to 
review it? Thanks.

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
> Attachments: Screen Shot 2017-10-16 at 17.27.48.png
>
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206852#comment-16206852
 ] 

Hadoop QA commented on YARN-7244:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m  9s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m 
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
3m 55s{color} | {color:orange} root: The patch generated 1 new + 318 unchanged 
- 2 fixed = 319 total (was 320) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
52s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. 
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red}  0m 
47s{color} | {color:red} hadoop-mapreduce-client-shuffle in the patch failed. 
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 47s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
47s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
 generated 7 new + 103 unchanged - 0 fixed = 110 total (was 103) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
4s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 46s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 43s{color} 
| {color:red} hadoop-mapreduce-client-shuffle in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
45s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 33s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7244 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892477/YARN-7244.010.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall

[jira] [Updated] (YARN-7197) Add support for a volume blacklist for docker containers

2017-10-16 Thread Eric Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7197:

Attachment: YARN-7197.001.patch

Basic check for black list requires exact match of path to ban.  The evaluation 
order is white list first, then black list.  This is assuming majority of the 
volumes can be mounted, and there are only a few system protected area, which 
can not be mounted.

> Add support for a volume blacklist for docker containers
> 
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Shane Kumpf
> Attachments: YARN-7197.001.patch
>
>
> Docker supports bind mounting host directories into containers. Work is 
> underway to allow admins to configure a whilelist of volume mounts. While 
> this is a much needed and useful feature, it opens the door for 
> misconfiguration that may lead to users being able to compromise or crash the 
> system. 
> One example would be allowing users to mount /run from a host running 
> systemd, and then running systemd in that container, rendering the host 
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist 
> would be where we put files and directories that if mounted into a container, 
> are likely to have negative consequences. Users are encouraged not to remove 
> items from the default blacklist, but may do so if necessary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-7197) Add support for a volume blacklist for docker containers

2017-10-16 Thread Eric Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang reassigned YARN-7197:
---

Assignee: Eric Yang

> Add support for a volume blacklist for docker containers
> 
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch
>
>
> Docker supports bind mounting host directories into containers. Work is 
> underway to allow admins to configure a whilelist of volume mounts. While 
> this is a much needed and useful feature, it opens the door for 
> misconfiguration that may lead to users being able to compromise or crash the 
> system. 
> One example would be allowing users to mount /run from a host running 
> systemd, and then running systemd in that container, rendering the host 
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist 
> would be where we put files and directories that if mounted into a container, 
> are likely to have negative consequences. Users are encouraged not to remove 
> items from the default blacklist, but may do so if necessary.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7202) Add UT for api-server

2017-10-16 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206843#comment-16206843
 ] 

Gour Saha commented on YARN-7202:
-

I think these 2 changes steers us away from REST style and hence should be 
reverted -
{noformat}
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java
@@ -165,12 +165,12 @@ private Response stopService(String appName, boolean 
destroy) {
   } else {
 LOG.info("Successfully stopped service {}", appName);
   }
-  return Response.status(Status.NO_CONTENT).build();
+  return Response.status(Status.OK).build();
 } catch (ApplicationNotFoundException e) {
   ServiceStatus serviceStatus = new ServiceStatus();
   serviceStatus.setDiagnostics(
-  "Service " + appName + " not found " + e.getMessage());
-  return Response.status(Status.NOT_FOUND).entity(serviceStatus)
+  "Service " + appName + " is not found in YARN: " + e.getMessage());
+  return Response.status(Status.BAD_REQUEST).entity(serviceStatus)
   .build();
 } catch (Exception e) {
   ServiceStatus serviceStatus = new ServiceStatus();
{noformat}

1. NO_CONTENT is pretty much similar to OK except that that the server will not 
return a response entity-body. Returning OK tells the client that they should 
expect a resource in the response as well, which is what we are NOT doing. So 
this needs to be reverted back to NO_CONTENT.

2. BAD_REQUEST means that the server did not understand the request, typically 
due to malformed syntax. It also hints that the client should not repeat the 
request (without modification). NOT_FOUND is the most appropriate response code 
here since it signifies that the service does not exist 
(ApplicationNotFoundException). The same request could possibly work later if a 
service of the same name was created down the line. So this needs to be 
reverted back to NOT_FOUND.

> Add UT for api-server
> -
>
> Key: YARN-7202
> URL: https://issues.apache.org/jira/browse/YARN-7202
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Eric Yang
> Fix For: yarn-native-services
>
> Attachments: YARN-7202.yarn-native-services.001.patch, 
> YARN-7202.yarn-native-services.002.patch, 
> YARN-7202.yarn-native-services.003.patch, 
> YARN-7202.yarn-native-services.004.patch, 
> YARN-7202.yarn-native-services.005.patch, 
> YARN-7202.yarn-native-services.006.patch, 
> YARN-7202.yarn-native-services.007.patch, 
> YARN-7202.yarn-native-services.008.patch, 
> YARN-7202.yarn-native-services.011.patch, 
> YARN-7202.yarn-native-services.012.patch, 
> YARN-7202.yarn-native-services.013.patch, 
> YARN-7202.yarn-native-services.014.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206841#comment-16206841
 ] 

Yufei Gu commented on YARN-4859:


Uploaded the screen shot of my tests. It shows how I test reservation. Please 
chime in if you have anything. [~asuresh] and [~subru]. 

BTW, there are some bugs on the webUI. For examples, queue name should just be 
"root.queueA" instead of internal queue name. All metrics(Allocated CPU, % of 
queue, etc) are missing for reservation runs. These should be a blocker though, 
filed YARN-7342 for this.

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
> Attachments: Screen Shot 2017-10-16 at 17.27.48.png
>
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7342) Application page doesn't show correct metrics for reservation runs

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7342:
---
Description: As the screen shot shows, there are some bugs on the webUI 
while running job with reservation. For examples, queue name should just be 
"root.queueA" instead of internal queue name. All metrics(Allocated CPU, % of 
queue, etc) are missing for reservation runs. These should be a blocker though. 

> Application page doesn't show correct metrics for reservation runs 
> ---
>
> Key: YARN-7342
> URL: https://issues.apache.org/jira/browse/YARN-7342
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler, reservation system
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
> Attachments: Screen Shot 2017-10-16 at 17.27.48.png
>
>
> As the screen shot shows, there are some bugs on the webUI while running job 
> with reservation. For examples, queue name should just be "root.queueA" 
> instead of internal queue name. All metrics(Allocated CPU, % of queue, etc) 
> are missing for reservation runs. These should be a blocker though. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7342) Application page doesn't show correct metrics for reservation runs

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7342:
---
Attachment: Screen Shot 2017-10-16 at 17.27.48.png

> Application page doesn't show correct metrics for reservation runs 
> ---
>
> Key: YARN-7342
> URL: https://issues.apache.org/jira/browse/YARN-7342
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler, reservation system
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
> Attachments: Screen Shot 2017-10-16 at 17.27.48.png
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7342) Application page doesn't show correct metrics for reservation runs

2017-10-16 Thread Yufei Gu (JIRA)

Yufei Gu created YARN-7342:
--

 Summary: Application page doesn't show correct metrics for 
reservation runs 
 Key: YARN-7342
 URL: https://issues.apache.org/jira/browse/YARN-7342
 Project: Hadoop YARN
  Issue Type: Bug
  Components: fairscheduler, reservation system
Affects Versions: 3.1.0
Reporter: Yufei Gu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206837#comment-16206837
 ] 

Hadoop QA commented on YARN-7169:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m 
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5355_branch2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m 
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
56s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 
32s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
 9s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 19m 
49s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn . 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  8m 
23s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 12m 
11s{color} | {color:green} YARN-5355_branch2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m 
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
4m 13s{color} | {color:orange} root: The patch generated 3 new + 286 unchanged 
- 0 fixed = 289 total (was 286) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 2s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m 
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
1s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 53s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
44s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}265m 58s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
|

[jira] [Commented] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206836#comment-16206836
 ] 

Hadoop QA commented on YARN-7341:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 21s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m  8s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
3s{color} | {color:green} hadoop-yarn-server-router in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 52s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7341 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892492/YARN-7341.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 5c44187fc487 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / e906108 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/17976/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17976/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL:

[jira] [Updated] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-4859:
---
Attachment: Screen Shot 2017-10-16 at 17.27.48.png

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
> Attachments: Screen Shot 2017-10-16 at 17.27.48.png
>
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-4859:
---
Attachment: (was: Screen Shot 2017-10-16 at 17.27.48.png)

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-4859:
---
Attachment: Screen Shot 2017-10-16 at 17.27.48.png

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
> Attachments: Screen Shot 2017-10-16 at 17.27.48.png
>
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7339) LocalityMulticastAMRMProxyPolicy should handle cancel request properly

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206829#comment-16206829
 ] 

Hadoop QA commented on YARN-7339:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 34s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red}  0m 
41s{color} | {color:red} hadoop-yarn-server-common in the patch failed. {color} 
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 32s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
30s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 56s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7339 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892489/YARN-7339-v1.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 1b0113485240 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 8a61525 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvninstall | 
https://builds.apache.org/job/PreCommit-YARN-Build/17975/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/17975/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17975/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically

[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206827#comment-16206827
 ] 

Hadoop QA commented on YARN-7217:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 32 new or modified test 
files. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
38s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 23m 
21s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
42s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
36s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
20m  5s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  2m  
6s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
yarn-native-services has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m  
2s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
58s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 20m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 20m 
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 56s{color} | {color:orange} root: The patch generated 14 new + 235 unchanged 
- 8 fixed = 249 total (was 243) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m 
21s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 34s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  9m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
19s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
17s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 35s{color} 
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 34s{color} 
| {color:red} hadoop-yarn-services-api in the patch

[jira] [Commented] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206825#comment-16206825
 ] 

Yufei Gu commented on YARN-4859:


[~asuresh], thanks for the explanation. I tested either drf and fair policy. 
Both works. The trick is to set reservation-request vcores to zero for fair 
policy queues. This is due to the reservation calculation is based on steady 
fair share, whose vcores always be zero for fair policy queues. We could do 
some improvement, like ignoring the vcores if the request is for fair policy 
queues. Probably create another JIRA for this.
[~subru], I tested creating reservation and submitting a job with the 
reservation id, it works without YARN-7311. We still need to fix the 
"root.root" queue issue though. I'll post the patch later. Besides that, 
YARN-7311 should be considered as a blocker as well. Could you spare time to 
review it? Thanks.

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206815#comment-16206815
 ] 

Hadoop QA commented on YARN-4511:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 6 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  6m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
49s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
10s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
52s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
7s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  9s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
50s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
54s{color} | {color:green} root: The patch generated 0 new + 534 unchanged - 2 
fixed = 534 total (was 536) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 22s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
20s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
49s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 7 new + 297 unchanged - 0 fixed = 304 total (was 297) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
41s{color} | {color:red} hadoop-tools_hadoop-sls generated 4 new + 20 unchanged 
- 0 fixed = 24 total (was 20) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 29s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 21s{color} 
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}102m 30s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Increment of volatile field 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode.numGuaranteedContainers
 in

[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

2017-10-16 Thread Eric Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-6623:

Attachment: cetest.stderr
cetest.stdout

[~vvasudev] Here are the output of the failed test.  Any idea why the tests 
fail?

> Add support to turn off launching privileged containers in the 
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch, 
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, 
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, 
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, 
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, 
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, 
> YARN-6623.013.patch, cetest.stderr, cetest.stdout
>
>
> Currently, launching privileged containers is controlled by the NM. We should 
> add a flag to the container-executor.cfg allowing admins to disable launching 
> privileged containers at the container-executor level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206797#comment-16206797
 ] 

Hadoop QA commented on YARN-7286:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 48s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  5m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  5m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 10s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
43s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m  7s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 57s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7286 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892476/YARN-7286.006.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  xml  findbugs  checkstyle  |
| uname | Linux 6926839372a6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 4540ffd |
| Default Java |

[jira] [Updated] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey

2017-10-16 Thread Robert Kanter (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated YARN-7341:

Attachment: YARN-7341.001.patch

It turns out that this is a real bug introduced by YARN-7095.  
{{RouterWebServiceUtil#mergeMetrics}} takes into two sets of metrics and merges 
them into the first one.  However, for a number of the metrics, it actually 
simply doubles the first metric.  For example
{code:java}
metrics.setTotalNodes(metrics.getTotalNodes() + metrics.getTotalNodes());
{code}
should be
{code:java}
metrics.setTotalNodes(metrics.getTotalNodes() + 
metricsResponse.getTotalNodes());
{code}

This should have failed every time, but the test also had a "flaw", which only 
made it flakey.  The test initializes two sets of metrics to random values 
using different {{Random}} objects using {{System.getCurrentTimeMillis()}} for 
the seed.  However, the code is fast enough that it often takes less than 1ms, 
causing the two objects to use the same seed.  When this happens, the two sets 
of metrics have the same values, and will mask the bug I described earlier.  If 
the code is slower (e.g. GC pause, swapping, adding a log statement for the 
seed, etc), then you'll get different seed values and the test will (correctly) 
fail.

The 001 patch fixes the bug by using the correct metric in 
{{RouterWebServiceUtil#mergeMetrics}}.  And it fixes the test by ensuring that 
the two seeds will be different.  It also cleans up some formatting and logs 
the seed for better debugability.

> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL: https://issues.apache.org/jira/browse/YARN-7341
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7341.001.patch
>
>
> {{TestRouterWebServiceUtil#testMergeMetrics}} is flakey.  It sometimes fails 
> with something like:
> {noformat}
> Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<< 
> FAILURE! - in 
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
>   Time elapsed: 0.005 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<1092> but was:<584>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7323) Data structure update in service REST API

2017-10-16 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206787#comment-16206787
 ] 

Gour Saha commented on YARN-7323:
-

{quote}
Data structure for Yarnfile organized launched_command and state at Service 
level, but the grouping definition was following Hadoop terminology. For 
clarity, we will change data structure to match Hadoop terminology. State and 
launch commands are moved to component level.
{quote}
[~eyang], launch command was always there at component-level. It was there at 
the service-level as well to support simple no-component services. This patch 
is killing the concept of simple no-component service and forcing the user to 
create a component even for the simplest case. Also, state is still there at 
the service-level along with component-level. Hence, the above description does 
not match with what the assignee of this Jira and the author of the patch 
intends to do. Next time, please consult with the assignee/patch-author before 
making changes to the description OR make your suggestions as comments rather 
than re-writing description OR request for ownership change of the jira and 
then make changes to the description and provide your patch.


> Data structure update in service REST API
> -
>
> Key: YARN-7323
> URL: https://issues.apache.org/jira/browse/YARN-7323
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7323.yarn-native-services.01.patch, 
> YARN-7323.yarn-native-services.02.patch, 
> YARN-7323.yarn-native-services.03.patch
>
>
> Unix terminology implies daemon process, such as httpd as a service.  In 
> Hadoop terminology, hdfs is a file system service, and hdfs has various 
> components such as namenode, journal nodes, and datanodes.  YARN service data 
> model was more closely describing daemon process as a service.  This JIRA 
> propose to remap the YARN service data model to align with Hadoop 
> terminology.  Service is composed of a collection of components.  Component 
> is either a unix process or docker container.
>   
> Data structure for Yarnfile organized launched_command and state at Service 
> level, but the grouping definition was following Hadoop terminology.  For 
> clarity, we will change data structure to match Hadoop terminology.  State 
> and launch commands are moved to component level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey

2017-10-16 Thread Robert Kanter (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter updated YARN-7341:

Affects Version/s: (was: 2.9.0)
 Target Version/s: 3.0.0  (was: 2.9.0, 3.0.0)

> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL: https://issues.apache.org/jira/browse/YARN-7341
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>
> {{TestRouterWebServiceUtil#testMergeMetrics}} is flakey.  It sometimes fails 
> with something like:
> {noformat}
> Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<< 
> FAILURE! - in 
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
>   Time elapsed: 0.005 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<1092> but was:<584>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey

2017-10-16 Thread Robert Kanter (JIRA)

Robert Kanter created YARN-7341:
---

 Summary: TestRouterWebServiceUtil#testMergeMetrics is flakey
 Key: YARN-7341
 URL: https://issues.apache.org/jira/browse/YARN-7341
 Project: Hadoop YARN
  Issue Type: Bug
  Components: federation
Affects Versions: 3.0.0-beta1, 2.9.0
Reporter: Robert Kanter
Assignee: Robert Kanter


{{TestRouterWebServiceUtil#testMergeMetrics}} is flakey.  It sometimes fails 
with something like:
{noformat}
Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<< 
FAILURE! - in 
org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
  Time elapsed: 0.005 sec  <<< FAILURE!
java.lang.AssertionError: expected:<1092> but was:<584>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at 
org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups

2017-10-16 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206771#comment-16206771
 ] 

Jonathan Hung edited comment on YARN-6852 at 10/16/17 11:30 PM:


Hi [~leftnoteasy], in {{gpu-module.c#internal_handle_gpu_request}}, there's 
this code: {noformat}+  // Use cgroup helpers to blacklist devices
+  for (int i = 0; i < n_minor_devices_to_block; i++) {
+char param_value[128];
+memset(param_value, 0, sizeof(param_value));
+snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, i);
+
+int rc = update_cgroups_parameters_func_p("devices", "deny",
+  container_id, param_value);
+
+if (0 != rc) {
+  fprintf(ERRORFILE, "CGroups: Failed to update cgroups\n");
+  return_code = -1;
+  goto cleanup;
+}
+  }{noformat}

Is {noformat}+snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, i);{noformat} supposed to be {noformat}+
snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, minor_devices[i]);{noformat}? Seems the 
minor number is not actually the number being written.

If so, I'll create a YARN-6223 subtask to address this.


was (Author: jhung):
Hi [~leftnoteasy], in {{gpu-module.c#internal_handle_gpu_request}}, there's 
this code: {noformat}+  // Use cgroup helpers to blacklist devices
+  for (int i = 0; i < n_minor_devices_to_block; i++) {
+char param_value[128];
+memset(param_value, 0, sizeof(param_value));
+snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, i);
+
+int rc = update_cgroups_parameters_func_p("devices", "deny",
+  container_id, param_value);
+
+if (0 != rc) {
+  fprintf(ERRORFILE, "CGroups: Failed to update cgroups\n");
+  return_code = -1;
+  goto cleanup;
+}
+  }{noformat}

Is {noformat}+snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, i);{noformat} supposed to be {noformat}+
snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, minor_devices[i]);{noformat}? Seems the 
minor number is not actually the number being written.

> [YARN-6223] Native code changes to support isolate GPU devices by using 
> CGroups
> ---
>
> Key: YARN-6852
> URL: https://issues.apache.org/jira/browse/YARN-6852
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6852.001.patch, YARN-6852.002.patch, 
> YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, 
> YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, 
> YARN-6852.009.patch
>
>
> This JIRA plan to add support of:
> 1) Isolation in CGroups. (native side).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly

2017-10-16 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206773#comment-16206773
 ] 

Gour Saha commented on YARN-7217:
-

{quote}
For changing number of containers with in a service, the api call must invoke:
1. Update service configuration
2. Stop existing service
3. Start service
{quote}
Are you saying that the API will not support flexing of no of containers when 
the service is running?

{quote}
The patch allows the configuration to be stored in Solr for faster search and 
retrieval time. There is a on/off flag to use Slor as backend. When solr 
storage backend is disabled (default), it will look for spec in HDFS.
{quote}
[~eyang], the description of the jira says "PUT method for update service for 
Service API doesn't function correctly". It is about functionality and not 
performance. Please file a separate jira for performance & fast search and move 
the Solr changes there as a separate patch. Additionally, the patch for 
YARN-7216 contains the exact same Solr code that is here in this patch as well, 
which makes it even more confusing.

> PUT method for update service for Service API doesn't function correctly
> 
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch, 
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object 
> contains state: STARTED.  The user would like to increase number of 
> containers for the deployed service.  The JSON has been updated to increase 
> container count.  The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object 
> contains state: STOPPED.  The user would like to make a environment 
> configuration change.  The configuration does not get updated after PUT 
> method.
> This is possible to address by rearranging the logic of START/STOP after 
> configuration update.  However, there are other potential combinations that 
> can break PUT method.  For example, user like to make configuration changes, 
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for 
> configuration vs status.  This increase the number of action that can be 
> performed.  New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
>   "name":"[service_name]",
>   "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
>   "name": "[service_name]",
>   "state": "STOPPED|STARTED"
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups

2017-10-16 Thread Jonathan Hung (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206771#comment-16206771
 ] 

Jonathan Hung commented on YARN-6852:
-

Hi [~leftnoteasy], in {{gpu-module.c#internal_handle_gpu_request}}, there's 
this code: {noformat}+  // Use cgroup helpers to blacklist devices
+  for (int i = 0; i < n_minor_devices_to_block; i++) {
+char param_value[128];
+memset(param_value, 0, sizeof(param_value));
+snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, i);
+
+int rc = update_cgroups_parameters_func_p("devices", "deny",
+  container_id, param_value);
+
+if (0 != rc) {
+  fprintf(ERRORFILE, "CGroups: Failed to update cgroups\n");
+  return_code = -1;
+  goto cleanup;
+}
+  }{noformat}

Is {noformat}+snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, i);{noformat} supposed to be {noformat}+
snprintf(param_value, sizeof(param_value), "c %d:%d rwm",
+ major_device_number, minor_devices[i]);{noformat}? Seems the 
minor number is not actually the number being written.

> [YARN-6223] Native code changes to support isolate GPU devices by using 
> CGroups
> ---
>
> Key: YARN-6852
> URL: https://issues.apache.org/jira/browse/YARN-6852
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Fix For: 3.0.0-beta1
>
> Attachments: YARN-6852.001.patch, YARN-6852.002.patch, 
> YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, 
> YARN-6852.006.patch, YARN-6852.007.patch, YARN-6852.008.patch, 
> YARN-6852.009.patch
>
>
> This JIRA plan to add support of:
> 1) Isolation in CGroups. (native side).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7340) Missing the time stamp in exception message in Class NoOverCommitPolicy

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7340:
---
Description: 
It could be easily figured out by reading code.
{code}
  throw new ResourceOverCommitException(
  "Resources at time " + " would be overcommitted by "
  + "accepting reservation: " + reservation.getReservationId());
{code}

  was:
It could be easily figure out by reading code.
{code}
  throw new ResourceOverCommitException(
  "Resources at time " + " would be overcommitted by "
  + "accepting reservation: " + reservation.getReservationId());
{code}


> Missing the time stamp in exception message in Class NoOverCommitPolicy
> ---
>
> Key: YARN-7340
> URL: https://issues.apache.org/jira/browse/YARN-7340
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: reservation system
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Priority: Minor
>  Labels: newbie++
>
> It could be easily figured out by reading code.
> {code}
>   throw new ResourceOverCommitException(
>   "Resources at time " + " would be overcommitted by "
>   + "accepting reservation: " + reservation.getReservationId());
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7340) Missing the time stamp in exception message in Class NoOverCommitPolicy

2017-10-16 Thread Yufei Gu (JIRA)

Yufei Gu created YARN-7340:
--

 Summary: Missing the time stamp in exception message in Class 
NoOverCommitPolicy
 Key: YARN-7340
 URL: https://issues.apache.org/jira/browse/YARN-7340
 Project: Hadoop YARN
  Issue Type: Bug
  Components: reservation system
Affects Versions: 3.1.0
Reporter: Yufei Gu
Priority: Minor


It could be easily figure out by reading code.
{code}
  throw new ResourceOverCommitException(
  "Resources at time " + " would be overcommitted by "
  + "accepting reservation: " + reservation.getReservationId());
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7340) Missing the time stamp in exception message in Class NoOverCommitPolicy

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7340:
---
Labels: newbie++  (was: )

> Missing the time stamp in exception message in Class NoOverCommitPolicy
> ---
>
> Key: YARN-7340
> URL: https://issues.apache.org/jira/browse/YARN-7340
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: reservation system
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Priority: Minor
>  Labels: newbie++
>
> It could be easily figure out by reading code.
> {code}
>   throw new ResourceOverCommitException(
>   "Resources at time " + " would be overcommitted by "
>   + "accepting reservation: " + reservation.getReservationId());
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

2017-10-16 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206764#comment-16206764
 ] 

Eric Yang commented on YARN-6623:
-

The current trunk seems to fail on these test cases:

{code}
[  FAILED  ] 3 tests, listed below:
[  FAILED  ] TestDockerUtil.test_check_mount_permitted
[  FAILED  ] TestDockerUtil.test_normalize_mounts
[  FAILED  ] TestDockerUtil.test_add_rw_mounts
{code}


> Add support to turn off launching privileged containers in the 
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch, 
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, 
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, 
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, 
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, 
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, 
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should 
> add a flag to the container-executor.cfg allowing admins to disable launching 
> privileged containers at the container-executor level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

2017-10-16 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206761#comment-16206761
 ] 

Eric Yang commented on YARN-6623:
-

Talked to [~wangda] and he said the test can be triggered using:

{code}
-Dtest=cetest,test-container-executor -Pnative
{code}

To the maven command.  It would be nice if it is covered by default test 
without passing in test case name.

> Add support to turn off launching privileged containers in the 
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch, 
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, 
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, 
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, 
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, 
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, 
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should 
> add a flag to the container-executor.cfg allowing admins to disable launching 
> privileged containers at the container-executor level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7308) TestApplicationACLs fails with FairScheduler

2017-10-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206758#comment-16206758
 ] 

Hudson commented on YARN-7308:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13094 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13094/])
YARN-7308. TestApplicationACLs fails with FairScheduler (rkanter) (rkanter: rev 
8a615259281511de639c1f9aa719a842e34a523f)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ParameterizedSchedulerTestBase.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationACLs.java


> TestApplicationACLs fails with FairScheduler
> 
>
> Key: YARN-7308
> URL: https://issues.apache.org/jira/browse/YARN-7308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7308.001.patch, YARN-7308.002.patch
>
>
> {{TestApplicationACLs}} fails when using FairScheduler:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
> testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
>   Time elapsed: 94.563 sec  <<< FAILURE!
> java.lang.AssertionError: App State is not correct (timeout). 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
> {noformat}
> There's a bunch of messages like this in the output:
> {noformat}
> 2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
> (MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State 
> is : ACCEPTED Waiting for state : FAILED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7333) container-executor fails to remove entries from a directory that is not writable or executable

2017-10-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206757#comment-16206757
 ] 

Hudson commented on YARN-7333:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13094 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13094/])
YARN-7333. container-executor fails to remove entries from a directory 
(nroberts: rev 4540ffd15f1259d99a7847c60d44c51f29faa397)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c


> container-executor fails to remove entries from a directory that is not 
> writable or executable
> --
>
> Key: YARN-7333
> URL: https://issues.apache.org/jira/browse/YARN-7333
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1, 2.8.2
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Fix For: 2.9.0, 2.8.2, 2.8.3, 3.0.0
>
> Attachments: YARN-7333.001.patch, YARN-7333.002.patch
>
>
> Similar to the situation from YARN-4594, container-executor will fail to 
> cleanup directories that do not have write and execute permissions for the 
> directory.  YARN-4594 fixed the scenario where the directory is not readable, 
> but it missed the case where we can open the directory but either not 
> traverse it (i.e.: no execute permission) or cannot remove entries from 
> within it (i.e.: no write permissions).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7323) Data structure update in service REST API

2017-10-16 Thread Jian He (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-7323:
--
Attachment: YARN-7323.yarn-native-services.03.patch

Chatted with Gour, made two changes
- rename ServiceState#Ready to be stable to be consistent with ComponentState
- removed ComponentState#INIT, because it's a state that will never be seen.

> Data structure update in service REST API
> -
>
> Key: YARN-7323
> URL: https://issues.apache.org/jira/browse/YARN-7323
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7323.yarn-native-services.01.patch, 
> YARN-7323.yarn-native-services.02.patch, 
> YARN-7323.yarn-native-services.03.patch
>
>
> Unix terminology implies daemon process, such as httpd as a service.  In 
> Hadoop terminology, hdfs is a file system service, and hdfs has various 
> components such as namenode, journal nodes, and datanodes.  YARN service data 
> model was more closely describing daemon process as a service.  This JIRA 
> propose to remap the YARN service data model to align with Hadoop 
> terminology.  Service is composed of a collection of components.  Component 
> is either a unix process or docker container.
>   
> Data structure for Yarnfile organized launched_command and state at Service 
> level, but the grouping definition was following Hadoop terminology.  For 
> clarity, we will change data structure to match Hadoop terminology.  State 
> and launch commands are moved to component level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206750#comment-16206750
 ] 

Hadoop QA commented on YARN-7169:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5355_branch2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  8m 
 3s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
1s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
45s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m 
43s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
36s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
26s{color} | {color:green} YARN-5355_branch2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 56s{color} | {color:orange} root: The patch generated 3 new + 286 unchanged 
- 0 fixed = 289 total (was 286) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m  
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  6m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 45s{color} 
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 1s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}162m  0s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
|

[jira] [Updated] (YARN-7339) LocalityMulticastAMRMProxyPolicy should handle cancel request properly

2017-10-16 Thread Botong Huang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-7339:
---
Attachment: YARN-7339-v1.patch

> LocalityMulticastAMRMProxyPolicy should handle cancel request properly
> --
>
> Key: YARN-7339
> URL: https://issues.apache.org/jira/browse/YARN-7339
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
> Attachments: YARN-7339-v1.patch
>
>
> Currently inside AMRMProxy, LocalityMulticastAMRMProxyPolicy is not handling 
> and splitting cancel requests from AM properly: 
> # For node cancel request, we should not treat it as a localized resource 
> request. Otherwise it can lead to all weight zero issue when computing 
> localized resource weight. 
> # For ANY cancel, we should broadcast to all known subclusters, not just the 
> ones associated with localized resources. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7339) LocalityMulticastAMRMProxyPolicy should handle cancel request properly

2017-10-16 Thread Botong Huang (JIRA)

Botong Huang created YARN-7339:
--

 Summary: LocalityMulticastAMRMProxyPolicy should handle cancel 
request properly
 Key: YARN-7339
 URL: https://issues.apache.org/jira/browse/YARN-7339
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Botong Huang
Assignee: Botong Huang
Priority: Minor


Currently inside AMRMProxy, LocalityMulticastAMRMProxyPolicy is not handling 
and splitting cancel requests from AM properly: 

# For node cancel request, we should not treat it as a localized resource 
request. Otherwise it can lead to all weight zero issue when computing 
localized resource weight. 

# For ANY cancel, we should broadcast to all known subclusters, not just the 
ones associated with localized resources. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

2017-10-16 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206736#comment-16206736
 ] 

Eric Yang commented on YARN-6623:
-

[~vvasudev] How is the C++ test case getting triggered?  mvn clean test 
-Pnative doesn't trigger test_docker_util.cc to run.

> Add support to turn off launching privileged containers in the 
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Fix For: 3.0.0
>
> Attachments: YARN-6623-branch-2.013.patch, 
> YARN-6623-branch-2.014.patch, YARN-6623-branch-2.015.patch, 
> YARN-6623.001.patch, YARN-6623.002.patch, YARN-6623.003.patch, 
> YARN-6623.004.patch, YARN-6623.005.patch, YARN-6623.006.patch, 
> YARN-6623.007.patch, YARN-6623.008.patch, YARN-6623.009.patch, 
> YARN-6623.010.patch, YARN-6623.011.patch, YARN-6623.012.patch, 
> YARN-6623.013.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should 
> add a flag to the container-executor.cfg allowing admins to disable launching 
> privileged containers at the container-executor level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7333) container-executor fails to remove entries from a directory that is not writable or executable

2017-10-16 Thread Junping Du (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206731#comment-16206731
 ] 

Junping Du commented on YARN-7333:
--

Thanks [~jlowe] for the patch and [~nroberts] for review!

> container-executor fails to remove entries from a directory that is not 
> writable or executable
> --
>
> Key: YARN-7333
> URL: https://issues.apache.org/jira/browse/YARN-7333
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1, 2.8.2
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Fix For: 2.9.0, 2.8.2, 2.8.3, 3.0.0
>
> Attachments: YARN-7333.001.patch, YARN-7333.002.patch
>
>
> Similar to the situation from YARN-4594, container-executor will fail to 
> cleanup directories that do not have write and execute permissions for the 
> directory.  YARN-4594 fixed the scenario where the directory is not readable, 
> but it missed the case where we can open the directory but either not 
> traverse it (i.e.: no execute permission) or cannot remove entries from 
> within it (i.e.: no write permissions).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206730#comment-16206730
 ] 

Hadoop QA commented on YARN-7230:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 12m 
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
42s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
16s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:c2d96dd |
| JIRA Issue | YARN-7230 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892479/YARN-7230.branch-2.8.003.patch
 |
| Optional Tests |  asflicense  mvnsite  xml  |
| uname | Linux 2da28383a4ac 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2.8 / 602a884 |
| modules | C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site 
U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17973/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Document DockerContainerRuntime for branch-2.8 with proper scope and claim as 
> an experimental feature
> -
>
> Key: YARN-7230
> URL: https://issues.apache.org/jira/browse/YARN-7230
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.1
>Reporter: Junping Du
>Assignee: Shane Kumpf
>Priority: Blocker
>  Labels: ready-to-commit
> Attachments: YARN-7230.branch-2.8.001.patch, 
> YARN-7230.branch-2.8.002.patch, YARN-7230.branch-2.8.003.patch
>
>
> YARN-5258 is to document new feature for docker container runtime which 
> already get checked in trunk/branch-2. We need a similar one for branch-2.8. 
> However, given we missed several patches, we need to define narrowed scope of 
> these feature/improvements which match with existing patches landed in 2.8. 
> Also, like YARN-6622, to document it as experimental.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206680#comment-16206680
 ] 

Hadoop QA commented on YARN-7295:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 103 new or modified 
test files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
19s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  9m 
42s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m 
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  5m 
45s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m  
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red}  7m  2s{color} 
| {color:red} root generated 1 new + 1445 unchanged - 0 fixed = 1446 total (was 
1445) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
2m 55s{color} | {color:orange} root: The patch generated 134 new + 3552 
unchanged - 297 fixed = 3686 total (was 3849) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  9m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green}  0m 
 1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green}  0m 
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
2s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-project hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server . hadoop-yarn-project 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
 hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 17m 
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  5m 
56s{color} | {color:red} root generated 1 new + 53 unchanged - 0 fixed = 54 
total (was 53) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}527m 29s{color} 
|

[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Vrushali C (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206679#comment-16206679
 ] 

Vrushali C commented on YARN-7169:
--

Linking YARN-7338 as blocker for merge 

> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, 
> YARN-7169-YARN-5355_branch2.0001.patch, 
> YARN-7169-YARN-5355_branch2.0002.patch, 
> YARN-7169-YARN-5355_branch2.0003.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding 
> into Timeline Service v2's branch2 which is YARN-5355_branch2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7336) Unsafe cast from long to int Resource.hashCode() method

2017-10-16 Thread Andrew Wang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Wang updated YARN-7336:
--
Target Version/s: 3.0.0, 3.1.0  (was: 3.0.0-beta1, 3.1.0)

> Unsafe cast from long to int Resource.hashCode() method
> ---
>
> Key: YARN-7336
> URL: https://issues.apache.org/jira/browse/YARN-7336
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-beta1, 3.1.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
>
> For example:
> {code}
> final int prime = 47;
> long result = 0;
> for (ResourceInformation entry : resources) {
>   result = prime * result + entry.hashCode();
> }
> return (int) result;
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7338) Support same origin policy for cross site scripting prevention.

2017-10-16 Thread Vrushali C (JIRA)

Vrushali C created YARN-7338:


 Summary: Support same origin policy for cross site scripting 
prevention.
 Key: YARN-7338
 URL: https://issues.apache.org/jira/browse/YARN-7338
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn-ui-v2
Reporter: Vrushali C



Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new 
web UI) to branch2  
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E

--
Ui2 does not seem to support same origin policy for cross site scripting 
prevention.
The following parameters has no effect for /ui2:

hadoop.http.cross-origin.enabled = true
yarn.resourcemanager.webapp.cross-origin.enabled = true

This is because ui2 is designed as a separate web application.  WebFilters 
setup for existing resource manager doesn’t apply to the new web application.
Please open JIRA to track the security issue and resolve the problem prior to 
backporting this to branch-2.
This would minimize the risk to open up security hole in branch-2.

--



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature

2017-10-16 Thread Shane Kumpf (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf updated YARN-7230:
--
Attachment: YARN-7230.branch-2.8.003.patch

> Document DockerContainerRuntime for branch-2.8 with proper scope and claim as 
> an experimental feature
> -
>
> Key: YARN-7230
> URL: https://issues.apache.org/jira/browse/YARN-7230
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.1
>Reporter: Junping Du
>Assignee: Shane Kumpf
>Priority: Blocker
>  Labels: ready-to-commit
> Attachments: YARN-7230.branch-2.8.001.patch, 
> YARN-7230.branch-2.8.002.patch, YARN-7230.branch-2.8.003.patch
>
>
> YARN-5258 is to document new feature for docker container runtime which 
> already get checked in trunk/branch-2. We need a similar one for branch-2.8. 
> However, given we missed several patches, we need to define narrowed scope of 
> these feature/improvements which match with existing patches landed in 2.8. 
> Also, like YARN-6622, to document it as experimental.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206657#comment-16206657
 ] 

Hadoop QA commented on YARN-7230:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} branch-2.8 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  6m 
41s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} branch-2.8 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
15s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}  9m 15s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:c2d96dd |
| JIRA Issue | YARN-7230 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892466/YARN-7230.branch-2.8.002.patch
 |
| Optional Tests |  asflicense  mvnsite  xml  |
| uname | Linux 796c953890bc 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | branch-2.8 / 843b8b7 |
| whitespace | 
https://builds.apache.org/job/PreCommit-YARN-Build/17968/artifact/patchprocess/whitespace-eol.txt
 |
| modules | C: hadoop-project hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site 
U: . |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17968/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Document DockerContainerRuntime for branch-2.8 with proper scope and claim as 
> an experimental feature
> -
>
> Key: YARN-7230
> URL: https://issues.apache.org/jira/browse/YARN-7230
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.1
>Reporter: Junping Du
>Assignee: Shane Kumpf
>Priority: Blocker
>  Labels: ready-to-commit
> Attachments: YARN-7230.branch-2.8.001.patch, 
> YARN-7230.branch-2.8.002.patch
>
>
> YARN-5258 is to document new feature for docker container runtime which 
> already get checked in trunk/branch-2. We need a similar one for branch-2.8. 
> However, given we missed several patches, we need to define narrowed scope of 
> these feature/improvements which match with existing patches landed in 2.8. 
> Also, like YARN-6622, to document it as experimental.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7124) LogAggregationTFileController deletes/renames while file is open

2017-10-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206648#comment-16206648
 ] 

Hudson commented on YARN-7124:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13092 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13092/])
YARN-7124. LogAggregationTFileController deletes/renames while file is 
(junping_du: rev 1fcbe7cf5f2f7c609cd4158912f151848268ca9e)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/filecontroller/tfile/LogAggregationTFileController.java


> LogAggregationTFileController deletes/renames while file is open
> 
>
> Key: YARN-7124
> URL: https://issues.apache.org/jira/browse/YARN-7124
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.9.0, 3.0.0-beta1
>Reporter: Daryn Sharp
>Assignee: Jason Lowe
>Priority: Critical
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7124.001.patch
>
>
> YARN-6288 changes the log aggregation writer to be an autoclosable.  
> Unfortunately the try-with-resources block for the writer will either rename 
> or delete the log while open.
> Assuming the NM's behavior is correct, deleting open files only results in 
> ominous WARNs in the nodemanager log and increases the rate of logging in the 
> NN when the implicit try-with-resource close fails.  These red herrings 
> complicate debugging efforts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7308) TestApplicationACLs fails with FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206641#comment-16206641
 ] 

Yufei Gu commented on YARN-7308:


+1

> TestApplicationACLs fails with FairScheduler
> 
>
> Key: YARN-7308
> URL: https://issues.apache.org/jira/browse/YARN-7308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7308.001.patch, YARN-7308.002.patch
>
>
> {{TestApplicationACLs}} fails when using FairScheduler:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
> testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
>   Time elapsed: 94.563 sec  <<< FAILURE!
> java.lang.AssertionError: App State is not correct (timeout). 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
> {noformat}
> There's a bunch of messages like this in the output:
> {noformat}
> 2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
> (MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State 
> is : ACCEPTED Waiting for state : FAILED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7244) ShuffleHandler is not aware of disks that are added

2017-10-16 Thread Kuhu Shukla (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kuhu Shukla updated YARN-7244:
--
Attachment: YARN-7244.010.patch

Attaching revised patch that address review comments. Thanks a lot !

> ShuffleHandler is not aware of disks that are added
> ---
>
> Key: YARN-7244
> URL: https://issues.apache.org/jira/browse/YARN-7244
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: YARN-7244.001.patch, YARN-7244.002.patch, 
> YARN-7244.003.patch, YARN-7244.004.patch, YARN-7244.005.patch, 
> YARN-7244.006.patch, YARN-7244.007.patch, YARN-7244.008.patch, 
> YARN-7244.009.patch, YARN-7244.010.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM 
> startup. If disks later are added to the node then map tasks will start using 
> them but the ShuffleHandler will not be aware of them. The end result is that 
> the data cannot be shuffled from the node leading to fetch failures and 
> re-runs of the map tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7308) TestApplicationACLs fails with FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7308:
---
Description: 
{{TestApplicationACLs}} fails when using FairScheduler:
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
  Time elapsed: 94.563 sec  <<< FAILURE!
java.lang.AssertionError: App State is not correct (timeout). expected: 
but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
{noformat}

There's a bunch of messages like this in the output:
{noformat}
2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
(MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State is 
: ACCEPTED Waiting for state : FAILED
{noformat}

  was:
emphasized text{{TestApplicationACLs}} fails when using FairScheduler:
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
  Time elapsed: 94.563 sec  <<< FAILURE!
java.lang.AssertionError: App State is not correct (timeout). expected: 
but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
{noformat}

There's a bunch of messages like this in the output:
{noformat}
2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
(MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State is 
: ACCEPTED Waiting for state : FAILED
{noformat}


> TestApplicationACLs fails with FairScheduler
> 
>
> Key: YARN-7308
> URL: https://issues.apache.org/jira/browse/YARN-7308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7308.001.patch, YARN-7308.002.patch
>
>
> {{TestApplicationACLs}} fails when using FairScheduler:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
> testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
>   Time elapsed: 94.563 sec  <<< FAILURE!
> java.lang.AssertionError: App State is not correct (timeout). 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
> {noformat}
> There's a bunch of messages like this in the output:
> {noformat}
> 2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
> (MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State 
> is : ACCEPTED Waiting for state : FAILED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

2017-10-16 Thread Haibo Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206632#comment-16206632
 ] 

Haibo Chen commented on YARN-4511:
--

Patch updated to address the follow major issues:
1) rename SchedulerNode.launchedContainers to allocatedContainers to avoid 
confusion with the launchedOnNode tag.
2) rename allocationInThisHeart  to resourcesOfContainersPendingLaunch which is 
incremented when containers are allocated and decremented when containers are 
launched on the node. It is now detached from node heartbeat, should also be 
fine in the case of global scheduler.
3) replace two nodes passed to ContainerUpdateContext.swap() with a single 
node, given Arun's comment that container update is done locally on a node.

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7308) TestApplicationACLs fails with FairScheduler

2017-10-16 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-7308:
---
Description: 
emphasized text{{TestApplicationACLs}} fails when using FairScheduler:
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
  Time elapsed: 94.563 sec  <<< FAILURE!
java.lang.AssertionError: App State is not correct (timeout). expected: 
but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
{noformat}

There's a bunch of messages like this in the output:
{noformat}
2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
(MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State is 
: ACCEPTED Waiting for state : FAILED
{noformat}

  was:
{{TestApplicationACLs}} fails when using FairScheduler:
{noformat}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec <<< 
FAILURE! - in org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
  Time elapsed: 94.563 sec  <<< FAILURE!
java.lang.AssertionError: App State is not correct (timeout). expected: 
but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at 
org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
at 
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
{noformat}

There's a bunch of messages like this in the output:
{noformat}
2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
(MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State is 
: ACCEPTED Waiting for state : FAILED
{noformat}


> TestApplicationACLs fails with FairScheduler
> 
>
> Key: YARN-7308
> URL: https://issues.apache.org/jira/browse/YARN-7308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Attachments: YARN-7308.001.patch, YARN-7308.002.patch
>
>
> emphasized text{{TestApplicationACLs}} fails when using FairScheduler:
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 98.389 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
> testApplicationACLs(org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs)
>   Time elapsed: 94.563 sec  <<< FAILURE!
> java.lang.AssertionError: App State is not correct (timeout). 
> expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:284)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.verifyInvalidQueueWithAcl(TestApplicationACLs.java:422)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs.testApplicationACLs(TestApplicationACLs.java:186)
> {noformat}
> There's a bunch of messages like this in the output:
> {noformat}
> 2017-10-09 17:00:54,572 INFO  [main] resourcemanager.MockRM 
> (MockRM.java:waitForState(277)) - App : application_1507593559080_0006 State 
> is : ACCEPTED Waiting for state : FAILED
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7286) Add support for docker to have no capabilities

2017-10-16 Thread Eric Badger (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-7286:
--
Attachment: YARN-7286.006.patch

Attaching new patch to fix checkstyle

> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch, 
> YARN-7286.003.patch, YARN-7286.004.patch, YARN-7286.005.patch, 
> YARN-7286.006.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it 
> does not allow for the capabilities list to be NULL, since {{getStrings()}} 
> will treat an empty value the same as it treats an unset property. So, a NULL 
> list will actually give the default capabilities list.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7308) TestApplicationACLs fails with FairScheduler

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206628#comment-16206628
 ] 

Hadoop QA commented on YARN-7308:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 31s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 36s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 3 new + 27 unchanged - 7 fixed = 30 total (was 34) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 40s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 18s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
|   | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter |
|   | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
|   | 
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7308 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892448/YARN-7308.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 4e103fd51f1e 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 21bc855 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Vrushali C (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7295:
-
Attachment: (was: YARN-7295-branch-2.02.patch)

> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch, YARN-7295-branch-2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Varun Saxena (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-7295:
---
Attachment: YARN-7295-branch-2.02.patch

> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch, 
> YARN-7295-branch-2.02.patch, YARN-7295-branch-2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Varun Saxena (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-7295:
---
Attachment: (was: YARN-7295-YARN-5355_branch2.03.patch)

> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch, YARN-7295-branch-2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4511) Common scheduler changes supporting scheduler-specific implementations

2017-10-16 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-4511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-4511:
-
Attachment: YARN-4511-YARN-1011.04.patch

> Common scheduler changes supporting scheduler-specific implementations
> --
>
> Key: YARN-4511
> URL: https://issues.apache.org/jira/browse/YARN-4511
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Haibo Chen
> Attachments: YARN-4511-YARN-1011.00.patch, 
> YARN-4511-YARN-1011.01.patch, YARN-4511-YARN-1011.02.patch, 
> YARN-4511-YARN-1011.03.patch, YARN-4511-YARN-1011.04.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Varun Saxena (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Saxena updated YARN-7295:
---
Attachment: YARN-7295-YARN-5355_branch2.03.patch

As discussed offline, attaching a new patch

> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-YARN-5355_branch2.03.patch, 
> YARN-7295-branch-2.01.patch, YARN-7295-branch-2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Vrushali C (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7295:
-
Attachment: YARN-7295-branch-2.02.patch

> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch, YARN-7295-branch-2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5764) NUMA awareness support for launching containers

2017-10-16 Thread Devaraj K (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206622#comment-16206622
 ] 

Devaraj K commented on YARN-5764:
-

Thanks [~miklos.szeg...@cloudera.com] for the review.

bq. I see commented code in the body of the function and also in the unit tests.
I commented the recovery code and related tests code since the YARN-7033 was 
not committed by the time when the patch created.

bq. is package-info.java necessary?
It is necessary and it adds a checkstyle error if we don't have it.

I will update the patch with the comments fixed and uncommented code.

> NUMA awareness support for launching containers
> ---
>
> Key: YARN-5764
> URL: https://issues.apache.org/jira/browse/YARN-5764
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager, yarn
>Reporter: Olasoji
>Assignee: Devaraj K
> Attachments: NUMA Awareness for YARN Containers.pdf, NUMA Performance 
> Results.pdf, YARN-5764-v0.patch, YARN-5764-v1.patch, YARN-5764-v2.patch, 
> YARN-5764-v3.patch, YARN-5764-v4.patch, YARN-5764-v5.patch
>
>
> The purpose of this feature is to improve Hadoop performance by minimizing 
> costly remote memory accesses on non SMP systems. Yarn containers, on launch, 
> will be pinned to a specific NUMA node and all subsequent memory allocations 
> will be served by the same node, reducing remote memory accesses. The current 
> default behavior is to spread memory across all NUMA nodes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Vrushali C (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7295:
-
Attachment: (was: YARN-7295-branch2.02.patch)

> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206606#comment-16206606
 ] 

Hadoop QA commented on YARN-7295:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  7s{color} 
| {color:red} YARN-7295 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7295 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892460/YARN-7295-branch2.02.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17966/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch, YARN-7295-branch2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206605#comment-16206605
 ] 

Hadoop QA commented on YARN-7286:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  5s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 22s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
42s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 50s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
30s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 24s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7286 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892383/YARN-7286.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  xml  findbugs  checkstyle  |
| uname | Linux 49be850a9796 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |

[jira] [Commented] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature

2017-10-16 Thread Shane Kumpf (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206599#comment-16206599
 ] 

Shane Kumpf commented on YARN-7230:
---

Thanks, [~ebadger], yes we should. It looks like this is missing in trunk and 
branch-2 as well. I'll open a new issue to address that.

> Document DockerContainerRuntime for branch-2.8 with proper scope and claim as 
> an experimental feature
> -
>
> Key: YARN-7230
> URL: https://issues.apache.org/jira/browse/YARN-7230
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.1
>Reporter: Junping Du
>Assignee: Shane Kumpf
>Priority: Blocker
>  Labels: ready-to-commit
> Attachments: YARN-7230.branch-2.8.001.patch, 
> YARN-7230.branch-2.8.002.patch
>
>
> YARN-5258 is to document new feature for docker container runtime which 
> already get checked in trunk/branch-2. We need a similar one for branch-2.8. 
> However, given we missed several patches, we need to define narrowed scope of 
> these feature/improvements which match with existing patches landed in 2.8. 
> Also, like YARN-6622, to document it as experimental.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature

2017-10-16 Thread Shane Kumpf (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf updated YARN-7230:
--
Attachment: YARN-7230.branch-2.8.002.patch

> Document DockerContainerRuntime for branch-2.8 with proper scope and claim as 
> an experimental feature
> -
>
> Key: YARN-7230
> URL: https://issues.apache.org/jira/browse/YARN-7230
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 2.8.1
>Reporter: Junping Du
>Assignee: Shane Kumpf
>Priority: Blocker
>  Labels: ready-to-commit
> Attachments: YARN-7230.branch-2.8.001.patch, 
> YARN-7230.branch-2.8.002.patch
>
>
> YARN-5258 is to document new feature for docker container runtime which 
> already get checked in trunk/branch-2. We need a similar one for branch-2.8. 
> However, given we missed several patches, we need to define narrowed scope of 
> these feature/improvements which match with existing patches landed in 2.8. 
> Also, like YARN-6622, to document it as experimental.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5473) Expose per-application over-allocation info in the Resource Manager

2017-10-16 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-5473:
-
Summary: Expose per-application over-allocation info in the Resource 
Manager  (was: Expose over-allocation in the Resource Manager UI)

> Expose per-application over-allocation info in the Resource Manager
> ---
>
> Key: YARN-5473
> URL: https://issues.apache.org/jira/browse/YARN-5473
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Haibo Chen
>
> When enabling over-allocation of nodes, the resources in the cluster change. 
> We need to surface this information for users to understand these changes.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7337) Expose per-node over-allocation info in Resource Manager

2017-10-16 Thread Haibo Chen (JIRA)

Haibo Chen created YARN-7337:


 Summary: Expose per-node over-allocation info in Resource Manager
 Key: YARN-7337
 URL: https://issues.apache.org/jira/browse/YARN-7337
 Project: Hadoop YARN
  Issue Type: Task
  Components: resourcemanager
Affects Versions: 3.0.0-alpha3
Reporter: Haibo Chen
Assignee: Haibo Chen






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7337) Expose per-node over-allocation info in Resource Manager

2017-10-16 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-7337:
-
Issue Type: Sub-task  (was: Task)
Parent: YARN-1011

> Expose per-node over-allocation info in Resource Manager
> 
>
> Key: YARN-7337
> URL: https://issues.apache.org/jira/browse/YARN-7337
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha3
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7333) container-executor fails to remove entries from a directory that is not writable or executable

2017-10-16 Thread Nathan Roberts (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206580#comment-16206580
 ] 

Nathan Roberts commented on YARN-7333:
--

Thanks Jason. I'll commit this shortly.

> container-executor fails to remove entries from a directory that is not 
> writable or executable
> --
>
> Key: YARN-7333
> URL: https://issues.apache.org/jira/browse/YARN-7333
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1, 2.8.2
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-7333.001.patch, YARN-7333.002.patch
>
>
> Similar to the situation from YARN-4594, container-executor will fail to 
> cleanup directories that do not have write and execute permissions for the 
> directory.  YARN-4594 fixed the scenario where the directory is not readable, 
> but it missed the case where we can open the directory but either not 
> traverse it (i.e.: no execute permission) or cannot remove entries from 
> within it (i.e.: no write permissions).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7333) container-executor fails to remove entries from a directory that is not writable or executable

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206573#comment-16206573
 ] 

Hadoop QA commented on YARN-7333:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
25m  1s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 22s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m  7s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 57s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7333 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12892450/YARN-7333.002.patch |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux 9e711374b08c 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 21bc855 |
| Default Java | 1.8.0_144 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/17960/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/17960/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/17960/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> container-executor fails to remove entries from a directory that is not 
> writable or executable
> --
>
> Key: YARN-7333
> URL: https://issues.apache.org/jira/browse/YARN-7333
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha1, 2.8.2
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-7333.001.patch, YARN-7333.002.patch
>
>
> Similar to the situation from YARN-4594,

[jira] [Updated] (YARN-7217) PUT method for update service for Service API doesn't function correctly

2017-10-16 Thread Eric Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7217:

Attachment: YARN-7217.yarn-native-services.002.patch

- Rebase patch on current HEAD of yarn-native-services branch.

> PUT method for update service for Service API doesn't function correctly
> 
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch, 
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object 
> contains state: STARTED.  The user would like to increase number of 
> containers for the deployed service.  The JSON has been updated to increase 
> container count.  The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object 
> contains state: STOPPED.  The user would like to make a environment 
> configuration change.  The configuration does not get updated after PUT 
> method.
> This is possible to address by rearranging the logic of START/STOP after 
> configuration update.  However, there are other potential combinations that 
> can break PUT method.  For example, user like to make configuration changes, 
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for 
> configuration vs status.  This increase the number of action that can be 
> performed.  New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
>   "name":"[service_name]",
>   "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
>   "name": "[service_name]",
>   "state": "STOPPED|STARTED"
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7275) NM Statestore cleanup for Container updates

2017-10-16 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206568#comment-16206568
 ] 

Hudson commented on YARN-7275:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13091 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13091/])
YARN-7275. NM Statestore cleanup for Container updates. (Kartheek (arun suresh: 
rev a50be1b8f432f50c940b66d12c7de87b95ea47c0)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMNullStateStoreService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/TestNMLeveldbStateStoreService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerSchedulerEventType.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/RecoverPausedContainerLaunch.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerEventType.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/RecoveredContainerLaunch.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMMemoryStateStoreService.java


> NM Statestore cleanup for Container updates
> ---
>
> Key: YARN-7275
> URL: https://issues.apache.org/jira/browse/YARN-7275
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: kartheek muthyala
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7275.001.patch, YARN-7275.002.patch, 
> YARN-7275.003.patch, YARN-7275.004.patch, YARN-7275.005.patch, 
> YARN-7275.006.patch
>
>
> Currently, only resource updates are recorded in the NM state store, we need 
> to add ExecutionType updates as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added

2017-10-16 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206560#comment-16206560
 ] 

Jason Lowe commented on YARN-7244:
--

Thanks for updating the patch!

Nit: The whitespace separating the private static fields from the 
MockShuffleHandler definition was lost.

MockShuffleHandler2 should remain a static class.

Nit: it would be nice if the javadocs added in AuxiliaryService were formatted 
to fit the style of the other method docs in this class.

This change seems unnecessary:
{noformat}
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
@@ -224,8 +227,9 @@
   private NMTimelinePublisher nmMetricsPublisher;
 
   public ContainerManagerImpl(Context context, ContainerExecutor exec,
-  DeletionService deletionContext, NodeStatusUpdater nodeStatusUpdater,
-  NodeManagerMetrics metrics, LocalDirsHandlerService dirsHandler) {
+  DeletionService deletionContext,
+  NodeStatusUpdater nodeStatusUpdater, NodeManagerMetrics metrics,
+  LocalDirsHandlerService dirsHandler) {
 super(ContainerManagerImpl.class.getName());
 this.context = context;
 this.dirsHandler = dirsHandler;
 {noformat}

TestAuxServices.auxiliaryLocalPathHandler should be final and named like a 
constant, e.g.: MOCK_AUX_PATH_HANDLER

AuxiliaryLocalPathHandlerImpl should be package-private.

In TestContainerManager#testAuxPathHandler:
{code}
+try {
+  Path p = auxiliaryLocalPathHandler.getLocalPathForRead("test");
+  assertTrue(p != null);
+} catch (Exception e) {
+  fail("Unexpected Exception!" + e);
+}
{code}
This should just let the exception be thrown to fail the test directly rather 
than catch it explicitly only to fail in a generic way.

It would be less code to use a spy object on a real LocalDirsHandlerService 
that is initialized with the conf rather than a mock that calls real methods on 
an object without properly initializing it.

> ShuffleHandler is not aware of disks that are added
> ---
>
> Key: YARN-7244
> URL: https://issues.apache.org/jira/browse/YARN-7244
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: YARN-7244.001.patch, YARN-7244.002.patch, 
> YARN-7244.003.patch, YARN-7244.004.patch, YARN-7244.005.patch, 
> YARN-7244.006.patch, YARN-7244.007.patch, YARN-7244.008.patch, 
> YARN-7244.009.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM 
> startup. If disks later are added to the node then map tasks will start using 
> them but the ShuffleHandler will not be aware of them. The end result is that 
> the data cannot be shuffled from the node leading to fetch failures and 
> re-runs of the map tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7295) Backport ATSv2 to branch-2

2017-10-16 Thread Vrushali C (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7295:
-
Attachment: YARN-7295-branch2.02.patch


Attaching v002. I generated the patch as follows:

Checked out hadoop branch-2 and merged YARN-5355_branch2 onto it. 

{code} 
- git merge --no-ff origin/YARN-5355_branch2
- git diff --binary --no-prefix ^a596c6f5eb553cf38123d61b844696ddbec99be0 > 
../YARN-7295-branch2.02.patch

{code} 

The latest commit on branch2 at this point is: 

{code}
commit a596c6f5eb553cf38123d61b844696ddbec99be0
Author: Wei-Chiu Chuang 
Date:   Mon Oct 16 12:57:48 2017 -0700

HDFS-12485. expunge may fail to remove trash from encryption zone. 
Contributed by Wei-Chiu Chuang.

(cherry picked from commit 8dbc8909c92d502d10a7f94d1de3171878a43b04)
{code}



> Backport ATSv2 to branch-2
> --
>
> Key: YARN-7295
> URL: https://issues.apache.org/jira/browse/YARN-7295
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-7295-branch-2.01.patch, YARN-7295-branch2.02.patch
>
>
> This JIRA is to run QA on patch with ATSv2 changes on branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206554#comment-16206554
 ] 

Hadoop QA commented on YARN-7169:
-

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-YARN-Build/17965/console in case of 
problems.


> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, 
> YARN-7169-YARN-5355_branch2.0001.patch, 
> YARN-7169-YARN-5355_branch2.0002.patch, 
> YARN-7169-YARN-5355_branch2.0003.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding 
> into Timeline Service v2's branch2 which is YARN-5355_branch2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7275) NM Statestore cleanup for Container updates

2017-10-16 Thread Arun Suresh (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-7275:
--
Fix Version/s: 3.0.0
   2.9.0

> NM Statestore cleanup for Container updates
> ---
>
> Key: YARN-7275
> URL: https://issues.apache.org/jira/browse/YARN-7275
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: kartheek muthyala
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0
>
> Attachments: YARN-7275.001.patch, YARN-7275.002.patch, 
> YARN-7275.003.patch, YARN-7275.004.patch, YARN-7275.005.patch, 
> YARN-7275.006.patch
>
>
> Currently, only resource updates are recorded in the NM state store, we need 
> to add ExecutionType updates as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206540#comment-16206540
 ] 

Hadoop QA commented on YARN-7169:
-

(!) A patch to the testing environment has been detected. 
Re-executing against the patched versions to perform further tests. 
The console is at 
https://builds.apache.org/job/PreCommit-YARN-Build/17964/console in case of 
problems.


> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, 
> YARN-7169-YARN-5355_branch2.0001.patch, 
> YARN-7169-YARN-5355_branch2.0002.patch, 
> YARN-7169-YARN-5355_branch2.0003.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding 
> into Timeline Service v2's branch2 which is YARN-5355_branch2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Vrushali C (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7169:
-
Attachment: YARN-7169-YARN-5355_branch2.0004.patch

Uploading patch again to look at the whitespace warnings one more time 

> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, 
> YARN-7169-YARN-5355_branch2.0001.patch, 
> YARN-7169-YARN-5355_branch2.0002.patch, 
> YARN-7169-YARN-5355_branch2.0003.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding 
> into Timeline Service v2's branch2 which is YARN-5355_branch2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)

2017-10-16 Thread Vrushali C (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7169:
-
Attachment: (was: yarn-ui-screenshot.png)

> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, 
> YARN-7169-YARN-5355_branch2.0001.patch, 
> YARN-7169-YARN-5355_branch2.0002.patch, 
> YARN-7169-YARN-5355_branch2.0003.patch, 
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding 
> into Timeline Service v2's branch2 which is YARN-5355_branch2.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7013) merge related work for YARN-3926 branch

2017-10-16 Thread Daniel Templeton (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-7013:
---
Attachment: YARN-7013.branch-3.0.002.patch

Looks like the unit test failures were resolved by YARN-7270.  I added an 
addendum patch to reapply YARN-7270 on top of the backports.

> merge related work for YARN-3926 branch
> ---
>
> Key: YARN-7013
> URL: https://issues.apache.org/jira/browse/YARN-7013
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Sunil G
>Assignee: Daniel Templeton
> Attachments: YARN-7013.001.patch, YARN-7013.002.patch, 
> YARN-7013.003.patch, YARN-7013.004.patch, YARN-7013.005.patch, 
> YARN-7013.006.patch, YARN-7013.008.patch, YARN-7013.branch-3.0.000.patch, 
> YARN-7013.branch-3.0.001.patch, YARN-7013.branch-3.0.002.patch
>
>
> To run jenkins for whole branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Arun Suresh (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206521#comment-16206521
 ] 

Arun Suresh edited comment on YARN-4859 at 10/16/17 8:05 PM:
-

Sorry to chime in late [~yufeigu], but If you are testing this, please note: I 
don't know if it is documented anywhere, but reservations will only work in FS 
queues which are configured with DRF policy (alteast this was the case when I 
testing this last year)


was (Author: asuresh):
Sorry to chime in late [~yufeigu], but If you are testing this, please note: I 
don't know if it is documented anywhere, but reservations will work in FS 
queues which are configured with DRF policy (alteast this was the case when I 
testing this last year)

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Arun Suresh (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206521#comment-16206521
 ] 

Arun Suresh commented on YARN-4859:
---

Sorry to chime in late [~yufeigu], but If you are testing this, please note: I 
don't know if it is documented anywhere, but reservations will work in FS 
queues which are configured with DRF policy (alteast this was the case when I 
testing this last year)

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler

2017-10-16 Thread Subru Krishnan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206510#comment-16206510
 ] 

Subru Krishnan commented on YARN-4859:
--

Thanks for the clarification [~yufeigu]. I agree we should fix the "root.root" 
corner case.

bq. Otherwise, reservation queue seems work for me.

This is great to hear. One question - have you created a reservation and 
submitted to the reservation or simply to a reservable queue?

> [Bug] Unable to submit a job to a reservation when using FairScheduler
> --
>
> Key: YARN-4859
> URL: https://issues.apache.org/jira/browse/YARN-4859
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: fairscheduler
>Reporter: Subru Krishnan
>Assignee: Yufei Gu
>Priority: Blocker
>
> Jobs submitted to a reservation get stuck at scheduled stage when using 
> FairScheduler. I came across this when working on YARN-4827 (documentation 
> for configuring ReservationSystem for FairScheduler)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor

2017-10-16 Thread Hadoop QA (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206505#comment-16206505
 ] 

Hadoop QA commented on YARN-6623:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 17m 
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 13 new or modified test 
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  7m 
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
44s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
53s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
59s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
22s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  2m 
49s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 0 
unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
39s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 24 unchanged - 5 fixed = 24 total (was 29) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  3m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}104m 13s{color} 
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 58s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m  
7s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red}  0m 
21s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}176m 15s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
|   | hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
| Timed out junit tests | 
org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater |
|   | org.apache.hadoop.yarn.server.resourcemanager.TestRMHA |
\\
\\
||

1 2 >

1 - 100 of 170 matches

Mail list logo