date:20180413

[jira] [Commented] (YARN-8126) [Follow up] Support auto-spawning of admin configured services during bootstrap of rm

2018-04-13 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438240#comment-16438240
 ] 

Rohith Sharma K S commented on YARN-8126:
-

[~sunilg] Could you please help to commit this patch? And please use above 
command so that whitespace also will be fixed. 

> [Follow up] Support auto-spawning of admin configured services during 
> bootstrap of rm
> -
>
> Key: YARN-8126
> URL: https://issues.apache.org/jira/browse/YARN-8126
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Attachments: YARN-8126.001.patch, YARN-8126.002.patch
>
>
> YARN-8048 adds support auto-spawning of admin configured services during 
> bootstrap of rm. 
> This JIRA is to follow up some of the comments discussed in YARN-8048. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8156) Increase the default value of yarn.timeline-service.app-collector.linger-period.ms

2018-04-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438239#comment-16438239
 ] 

Hudson commented on YARN-8156:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13997 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13997/])
YARN-8156. Increase the default value of (rohithsharmaks: rev 
669eb7bdea34f26e9b9b8a2260ae4356791622e7)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestPerNodeTimelineCollectorsAuxService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md


> Increase the default value of 
> yarn.timeline-service.app-collector.linger-period.ms
> --
>
> Key: YARN-8156
> URL: https://issues.apache.org/jira/browse/YARN-8156
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-8156.001.patch, YARN-8156.002.patch
>
>
> The default value of *yarn.timeline-service.app-collector.linger-period.ms* 
> is 1 sec which is very low. In production cluster, we observe that most of 
> the entity information is not published because of 1sec linger period. 
> I would suggest to increase the default value to either 10sec or 30sec or 60 
> sec. As far as I have seen, 10sec should be sufficient since node heartbeat 
> period is 3 sec by default, all NM gets request to stop the application. 
>  cc:/ [~haibo.chen] [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8156) Increase the default value of yarn.timeline-service.app-collector.linger-period.ms

2018-04-13 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438233#comment-16438233
 ] 

Rohith Sharma K S commented on YARN-8156:
-

committing shortly

> Increase the default value of 
> yarn.timeline-service.app-collector.linger-period.ms
> --
>
> Key: YARN-8156
> URL: https://issues.apache.org/jira/browse/YARN-8156
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-8156.001.patch, YARN-8156.002.patch
>
>
> The default value of *yarn.timeline-service.app-collector.linger-period.ms* 
> is 1 sec which is very low. In production cluster, we observe that most of 
> the entity information is not published because of 1sec linger period. 
> I would suggest to increase the default value to either 10sec or 30sec or 60 
> sec. As far as I have seen, 10sec should be sufficient since node heartbeat 
> period is 3 sec by default, all NM gets request to stop the application. 
>  cc:/ [~haibo.chen] [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7781) Update YARN-Services-Examples.md to be in sync with the latest code

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438206#comment-16438206
 ] 

genericqa commented on YARN-7781:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
15s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 34s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 26s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
36s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
36s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 90m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-7781 |
| JIRA Patch URL |

[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438182#comment-16438182
 ] 

genericqa commented on YARN-8162:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 21s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 19s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
 The patch generated 2 new + 13 unchanged - 0 fixed = 15 total (was 13) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 47s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 
12s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 43s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8162 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12919038/YARN-8162.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 558cb4598ccb 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 995cba6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/20347/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/20347/testReport/ |
| Max. process+thread count | 407 (vs. ulimit of 1)

[jira] [Commented] (YARN-7810) TestDockerContainerRuntime test failures due to UID lookup of a non-existent user

2018-04-13 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438177#comment-16438177
 ] 

Eric Yang commented on YARN-7810:
-

[~jojochuang] Thank you for catching this with JDK 7.  [~shaneku...@gmail.com] 
Can you help to with addendum patch to change String.join() to a for loop to 
write to StringBuffer?

> TestDockerContainerRuntime test failures due to UID lookup of a non-existent 
> user
> -
>
> Key: YARN-7810
> URL: https://issues.apache.org/jira/browse/YARN-7810
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
> Attachments: YARN-7810-branch-2.001.patch, 
> YARN-7810-branch-3.0.001.patch, YARN-7810.001.patch, YARN-7810.002.patch
>
>
> YARN-7782 enabled the Docker runtime feature to remap the username to uid:gid 
> form for launching Docker containers. The feature does an {{id -u}} and {{id 
> -G}} to get the UID and GIDs. This fails with the test user, as that user 
> doesn't actually exist on the host.
> {code:java}
> [ERROR] 
> testContainerLaunchWithCustomNetworks(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime)
>   Time elapsed: 0.411 s  <<< ERROR!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
>  
> ExitCodeException exitCode=1: id: 'run_as_user': no such user
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.getUserIdInfo(DockerLinuxContainerRuntime.java:711)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:757)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks(TestDockerContainerRuntime.java:599){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8138) Add unit test to validate queue priority preemption works under node partition.

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438159#comment-16438159
 ] 

genericqa commented on YARN-8138:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m 
 0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 23s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 31s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 6 new + 17 unchanged - 0 fixed = 23 total (was 17) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 33s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m 
20s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 17s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8138 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12919019/YARN-8138.003.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 71a83ac32a1e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 995cba6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/20346/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/20346/testReport/ |
| Max. process+thread count | 886 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U:

[jira] [Updated] (YARN-7781) Update YARN-Services-Examples.md to be in sync with the latest code

2018-04-13 Thread Gour Saha (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha updated YARN-7781:

Attachment: YARN-7781.05.patch

> Update YARN-Services-Examples.md to be in sync with the latest code
> ---
>
> Key: YARN-7781
> URL: https://issues.apache.org/jira/browse/YARN-7781
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: YARN-7781.01.patch, YARN-7781.02.patch, 
> YARN-7781.03.patch, YARN-7781.04.patch, YARN-7781.05.patch
>
>
> Update YARN-Services-Examples.md to make the following additions/changes:
> 1. Add an additional URL and PUT Request JSON to support flex:
> Update to flex up/down the no of containers (instances) of a component of a 
> service
> PUT URL – http://localhost:8088/app/v1/services/hello-world
> PUT Request JSON
> {code}
> {
>   "components" : [ {
> "name" : "hello",
> "number_of_containers" : 3
>   } ]
> }
> {code}
> 2. Modify all occurrences of /ws/ to /app/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8162) Method DirectoryCollection#verifyDirUsingMkdir isn't needed anymore

2018-04-13 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-8162:
---
Description: First of all, DirectoryCollection#verifyDirUsingMkdir doesn't 
do what it claims to do, it creates a file but its Java doc says it create a 
dir. Because of that, it doesn't compatible with {{ReadWriteDiskValidator}} 
because the validator will throw an exception if it isn't dir. What's more, it 
is no longer needed after HADOOP-13738 (DiskChecker should perform some disk 
IO). It duplicates what HADOOP-13738 does. We should remove it.   (was: First 
of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it claims to 
do, it creates a file but its Java doc says it create a dir. What's more, it is 
no longer needed after HADOOP-13738 (DiskChecker should perform some disk IO). 
It duplicates what HADOOP-13738 does. We should remove it.)

> Method DirectoryCollection#verifyDirUsingMkdir isn't needed anymore
> ---
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it 
> claims to do, it creates a file but its Java doc says it create a dir. 
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}} 
> because the validator will throw an exception if it isn't dir. What's more, 
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some 
> disk IO). It duplicates what HADOOP-13738 does. We should remove it. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir

2018-04-13 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-8162:
---
Summary: Remove Method DirectoryCollection#verifyDirUsingMkdir  (was: 
Method DirectoryCollection#verifyDirUsingMkdir isn't needed anymore)

> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it 
> claims to do, it creates a file but its Java doc says it create a dir. 
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}} 
> because the validator will throw an exception if it isn't dir. What's more, 
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some 
> disk IO). It duplicates what HADOOP-13738 does. We should remove it. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8162) Method DirectoryCollection#verifyDirUsingMkdir isn't needed anymore

2018-04-13 Thread Yufei Gu (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-8162:
---
Attachment: YARN-8162.001.patch

> Method DirectoryCollection#verifyDirUsingMkdir isn't needed anymore
> ---
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it 
> claims to do, it creates a file but its Java doc says it create a dir. What's 
> more, it is no longer needed after HADOOP-13738 (DiskChecker should perform 
> some disk IO). It duplicates what HADOOP-13738 does. We should remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8151) Yarn RM Epoch should wrap around

2018-04-13 Thread Young Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438134#comment-16438134
 ] 

Young Chen commented on YARN-8151:
--

[~giovanni.fumarola] can you take a look? Thanks!

> Yarn RM Epoch should wrap around
> 
>
> Key: YARN-8151
> URL: https://issues.apache.org/jira/browse/YARN-8151
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Young Chen
>Assignee: Young Chen
>Priority: Major
> Attachments: YARN-8151.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8151) Yarn RM Epoch should wrap around

2018-04-13 Thread Young Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Young Chen updated YARN-8151:
-
Attachment: YARN-8151.01.patch

> Yarn RM Epoch should wrap around
> 
>
> Key: YARN-8151
> URL: https://issues.apache.org/jira/browse/YARN-8151
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Young Chen
>Assignee: Young Chen
>Priority: Major
> Attachments: YARN-8151.01.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-8162) Method DirectoryCollection#verifyDirUsingMkdir isn't needed anymore

2018-04-13 Thread Yufei Gu (JIRA)

Yufei Gu created YARN-8162:
--

 Summary: Method DirectoryCollection#verifyDirUsingMkdir isn't 
needed anymore
 Key: YARN-8162
 URL: https://issues.apache.org/jira/browse/YARN-8162
 Project: Hadoop YARN
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 3.1.0
Reporter: Yufei Gu
Assignee: Yufei Gu


First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it claims 
to do, it creates a file but its Java doc says it create a dir. What's more, it 
is no longer needed after HADOOP-13738 (DiskChecker should perform some disk 
IO). It duplicates what HADOOP-13738 does. We should remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7781) Update YARN-Services-Examples.md to be in sync with the latest code

2018-04-13 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438118#comment-16438118
 ] 

Gour Saha commented on YARN-7781:
-

[~billie.rinaldi] I made changes to the AM kerberos principal/keytab 
documentation as you suggested.

> Update YARN-Services-Examples.md to be in sync with the latest code
> ---
>
> Key: YARN-7781
> URL: https://issues.apache.org/jira/browse/YARN-7781
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: YARN-7781.01.patch, YARN-7781.02.patch, 
> YARN-7781.03.patch, YARN-7781.04.patch
>
>
> Update YARN-Services-Examples.md to make the following additions/changes:
> 1. Add an additional URL and PUT Request JSON to support flex:
> Update to flex up/down the no of containers (instances) of a component of a 
> service
> PUT URL – http://localhost:8088/app/v1/services/hello-world
> PUT Request JSON
> {code}
> {
>   "components" : [ {
> "name" : "hello",
> "number_of_containers" : 3
>   } ]
> }
> {code}
> 2. Modify all occurrences of /ws/ to /app/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7781) Update YARN-Services-Examples.md to be in sync with the latest code

2018-04-13 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438110#comment-16438110
 ] 

Gour Saha commented on YARN-7781:
-

[~eyang] I think ServiceState.FLEX is not even required to trigger the flex 
operation via REST. I filed YARN-8161 to get rid of it. I am not going to make 
any FLEX related changes in this patch and address it all in YARN-8161. One 
example regarding the flex API was missing which has been added in this patch 
(without any change to the enum name).

bq. In the example JSON, dependencies, node_attributes, node_partitions are not 
clear to the reader. Can we fill in the value or remove them for ease of 
reading?
Removed dependencies and added values for node attributes and node partitions

bq. ...docker ENTRY_POINT support..
For the ENTRY_POINT suggestion, can you please file a new sub-task. 
[~billie.rinaldi] has made a lot of changes to the configurations and examples 
in YARN-8060 and it would be good to do this separately and on top of her 
changes.

> Update YARN-Services-Examples.md to be in sync with the latest code
> ---
>
> Key: YARN-7781
> URL: https://issues.apache.org/jira/browse/YARN-7781
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: YARN-7781.01.patch, YARN-7781.02.patch, 
> YARN-7781.03.patch, YARN-7781.04.patch
>
>
> Update YARN-Services-Examples.md to make the following additions/changes:
> 1. Add an additional URL and PUT Request JSON to support flex:
> Update to flex up/down the no of containers (instances) of a component of a 
> service
> PUT URL – http://localhost:8088/app/v1/services/hello-world
> PUT Request JSON
> {code}
> {
>   "components" : [ {
> "name" : "hello",
> "number_of_containers" : 3
>   } ]
> }
> {code}
> 2. Modify all occurrences of /ws/ to /app/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7996) Allow user supplied Docker client configurations with YARN native services

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438069#comment-16438069
 ] 

genericqa commented on YARN-7996:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
 8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  3s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 14s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
53s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  0m 32s{color} 
| {color:red} hadoop-yarn-services-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
18s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
32s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 91m 21s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.service.TestApiServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce

[jira] [Assigned] (YARN-8089) Recover domain from backend for TimelineCollector.

2018-04-13 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen reassigned YARN-8089:


Assignee: Haibo Chen

> Recover domain from backend for TimelineCollector.
> --
>
> Key: YARN-8089
> URL: https://issues.apache.org/jira/browse/YARN-8089
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineservice
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
>
> In-memory domain information is volatile. Hence, we need to store domain in 
> the backend, and recover when the TimelineCollector comes up again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs

2018-04-13 Thread Haibo Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438059#comment-16438059
 ] 

Haibo Chen commented on YARN-7088:
--

{quote}I update the launchTime only if it is non zero so I guess we are fine 
here?
{quote}
Sorry, I misunderstood the patch. But I am still not sure if we should add 
schedulingwait time and total run time. If an application have more than one 
attempts, and the following attempt(s) spend some time waiting to be scheduled, 
then the two metrics are not accurate. Thus, I'm inclined to not add them.

Besides, I have a few more minor comments.

1) The change to remove the period in ApplicationReport.getName() javadoc is 
unnecessary. That's why checkstyle is warning about this.

2) ApplicationReport.setLaunchTime should be marked as Private & Unstable, 
whereas getLaunchTime should be annotated with Public & Unstable, following the 
conventions

3) ApplicationMetricsConstants.APP_ATT_LAUNCHED_EVENT_TYPE is unused.

4) The name of  AppRunningOnAppAttemptLaunchTransition is misleading. How about 
we rename it to AttemptLaunchedTransition?

5) RMAppImpl.setLaunchTime() is not needed I think.

> Fix application start time and add submit time to UIs
> -
>
> Key: YARN-7088
> URL: https://issues.apache.org/jira/browse/YARN-7088
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Abdullah Yousufi
>Assignee: Kanwaljeet Sachdev
>Priority: Major
> Attachments: YARN-7088.001.patch, YARN-7088.002.patch, 
> YARN-7088.003.patch, YARN-7088.004.patch, YARN-7088.005.patch, 
> YARN-7088.006.patch, YARN-7088.007.patch, YARN-7088.008.patch, 
> YARN-7088.009.patch, YARN-7088.010.patch, YARN-7088.011.patch, 
> YARN-7088.012.patch, YARN-7088.013.patch, YARN-7088.014.patch, 
> YARN-7088.015.patch
>
>
> Currently, the start time in the old and new UI actually shows the app 
> submission time. There should actually be two different fields; one for the 
> app's submission and one for its start, as well as the elapsed pending time 
> between the two.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8138) Add unit test to validate queue priority preemption works under node partition.

2018-04-13 Thread Zian Chen (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438010#comment-16438010
 ] 

Zian Chen commented on YARN-8138:
-

Thanks [~leftnoteasy] for the reminder. Quickly checked failed cases, not 
related. Fixed checkstyle issue and re-upload the patch.

> Add unit test to validate queue priority preemption works under node 
> partition.
> ---
>
> Key: YARN-8138
> URL: https://issues.apache.org/jira/browse/YARN-8138
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zian Chen
>Priority: Minor
> Attachments: YARN-8138.001.patch, YARN-8138.002.patch, 
> YARN-8138.003.patch
>
>
> Add unit test to validate queue priority preemption works under node 
> partition.
> Test configuration:
>  queue A (capacity=50, priority=1)
>  queue B (capacity=50, priority=2)
>  both have accessible-node-labels set to x
>  A.accessible-node-labels.x.capacity = 50
>  B.accessible-node-labels.x.capacity = 50
>  Along with this pre-emption related properties have been set.
> Test steps:
>  - Submit an application A1 to B, with am-container = container = 4096, no. 
> of containers = 4
>  - Submit an application A2 to A, with am-container = 1024, container = 2048, 
> no of containers = (NUM_NM-1)
>  - Kill application A1
>  - Submit an application A3 to B with am-container=container=5210, no. of 
> containers=NUM_NM
>  - Expectation is that containers are pre-empted from application A2 to A3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8138) Add unit test to validate queue priority preemption works under node partition.

2018-04-13 Thread Zian Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zian Chen updated YARN-8138:

Attachment: YARN-8138.003.patch

> Add unit test to validate queue priority preemption works under node 
> partition.
> ---
>
> Key: YARN-8138
> URL: https://issues.apache.org/jira/browse/YARN-8138
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Zian Chen
>Priority: Minor
> Attachments: YARN-8138.001.patch, YARN-8138.002.patch, 
> YARN-8138.003.patch
>
>
> Add unit test to validate queue priority preemption works under node 
> partition.
> Test configuration:
>  queue A (capacity=50, priority=1)
>  queue B (capacity=50, priority=2)
>  both have accessible-node-labels set to x
>  A.accessible-node-labels.x.capacity = 50
>  B.accessible-node-labels.x.capacity = 50
>  Along with this pre-emption related properties have been set.
> Test steps:
>  - Submit an application A1 to B, with am-container = container = 4096, no. 
> of containers = 4
>  - Submit an application A2 to A, with am-container = 1024, container = 2048, 
> no of containers = (NUM_NM-1)
>  - Kill application A1
>  - Submit an application A3 to B with am-container=container=5210, no. of 
> containers=NUM_NM
>  - Expectation is that containers are pre-empted from application A2 to A3



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7189) Container-executor doesn't remove Docker containers that error out early

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438003#comment-16438003
 ] 

genericqa commented on YARN-7189:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
37s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
24m  2s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 49s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m 
22s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
25s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 54m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5aaf88d |
| JIRA Issue | YARN-7189 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12919010/YARN-7189-branch-3.0.004.patch
 |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux 3301d48b9e8d 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.0 / dc01e32 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/20344/testReport/ |
| Max. process+thread count | 397 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/20344/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Container-executor doesn't remove Docker containers that error out early
> 
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch, 
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch, 
> YARN-7189-branch-3.0.003.patch,

[jira] [Commented] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers

2018-04-13 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438000#comment-16438000
 ] 

Eric Yang commented on YARN-8160:
-

Base on what I read from YARN-7973, the relaunch event is encapsulated in 
ContainerLauncher.  By passing parameter to ContainerLauncher event, it can 
trigger relaunch of container to happen.  Where YARN-7939 code have added a new 
transition for relaunch in ContainerImpl, and trigger a new type of relaunch 
event.  The alignment can happen if we surface YARN-7973 logic from 
ContainerLauncher and connect with the new transition event for doRelaunch.  It 
might help to refine relaunch logic and reuse some of the existing code.

> Yarn Service Upgrade: Support upgrade of service that use docker containers 
> 
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
>
> Ability to upgrade dockerized  yarn native services.
> Ref: YARN-5637



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-8042) Improve debugging on ATSv2 reader server

2018-04-13 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen reassigned YARN-8042:


Assignee: Haibo Chen

> Improve debugging on ATSv2 reader server
> 
>
> Key: YARN-8042
> URL: https://issues.apache.org/jira/browse/YARN-8042
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
>
> It's been inconvenient to debug issues that happened on the read path. 
> Typically, a query sent from a client is parsed into a TimelineReaderContext, 
> TimelineEntityFilters  and TimelineDataToRetrieve  which are independent of 
> the underlying backend storage implementations.    Then the general ATSv2 
> building blocks are then translated into a SCAN and GET query in HBase with 
> specified row keys and filters.
> To facilitate easy debugging, additional debug level logging messages 
> (ideally that can be enabled dynamically without restarting 
> TimelineReaderServer process) can be added at the boundary to narrow down the 
> scope of investigation.
> A good example of this is logging the scan or get query before it is sent to 
> HBase and the result after the query returns from HBase. YARN support folks 
> who are not necessarily HBase experts can present the debug messages to HBase 
> experts and get help. (I had to remotely connect to TimelineReaderServer, set 
> up breakpoints and get the hbase queries every time I suspect there is a bug 
> in HBase)
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8161) ServiceState FLEX should be removed

2018-04-13 Thread Gour Saha (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha updated YARN-8161:

Target Version/s: 3.2.0, 3.1.1  (was: 3.1.1)

> ServiceState FLEX should be removed
> ---
>
> Key: YARN-8161
> URL: https://issues.apache.org/jira/browse/YARN-8161
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Gour Saha
>Priority: Major
>
> ServiceState FLEX is not required to trigger flex up/down of containers and 
> should be removed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-8161) ServiceState FLEX should be removed

2018-04-13 Thread Gour Saha (JIRA)

Gour Saha created YARN-8161:
---

 Summary: ServiceState FLEX should be removed
 Key: YARN-8161
 URL: https://issues.apache.org/jira/browse/YARN-8161
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn-native-services
Affects Versions: 3.1.0
Reporter: Gour Saha


ServiceState FLEX is not required to trigger flex up/down of containers and 
should be removed



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7189) Container-executor doesn't remove Docker containers that error out early

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437990#comment-16437990
 ] 

genericqa commented on YARN-7189:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-3.0 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
 4s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
48s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} branch-3.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
29m 13s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m  7s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m 
51s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5aaf88d |
| JIRA Issue | YARN-7189 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12919001/YARN-7189-branch-3.0.003.patch
 |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux c10a86f0c0ed 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.0 / dc01e32 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/20343/testReport/ |
| Max. process+thread count | 410 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/20343/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Container-executor doesn't remove Docker containers that error out early
> 
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch, 
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch, 
> YARN-7189-branch-3.0.003.patch,

[jira] [Commented] (YARN-8104) Add API to fetch node to attribute mapping

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437984#comment-16437984
 ] 

genericqa commented on YARN-8104:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  2m  
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
53s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m 
14s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
39s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
26s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
YARN-3409 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
38s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 25m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 25m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 25m 
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  4m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 25s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
18s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m 
30s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 27m 
18s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
28s{color} | {color:green} hadoop-yarn-server-router in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}129m 21s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License

[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers

2018-04-13 Thread Gour Saha (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha updated YARN-8160:

Description: 
Ability to upgrade dockerized  yarn native services.

Ref: YARN-5637

  was:Ability to upgrade dockerized  yarn native services.


> Yarn Service Upgrade: Support upgrade of service that use docker containers 
> 
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
>
> Ability to upgrade dockerized  yarn native services.
> Ref: YARN-5637



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8160) Yarn Service Upgrade: Support upgrade of service that use docker containers

2018-04-13 Thread Chandni Singh (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chandni Singh updated YARN-8160:

Summary: Yarn Service Upgrade: Support upgrade of service that use docker 
containers   (was: Yarn Service Upgrade: Support upgrade of docker containers )

> Yarn Service Upgrade: Support upgrade of service that use docker containers 
> 
>
> Key: YARN-8160
> URL: https://issues.apache.org/jira/browse/YARN-8160
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
>
> Ability to upgrade dockerized  yarn native services.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437969#comment-16437969
 ] 

Eric Yang commented on YARN-8060:
-

[~billie.rinaldi] Thank you for the patch.  Readiness check works as 
advertised.  I ran into a scenario that net=host.  Two docker containers ran on 
the same host and trying to grab the same port.  Readiness checks for both 
containers were responded healthy by the first container that grab the port.  
This is basically a user error that user can work around by defining placement 
policy, or use multiple components to define different ports to prevent 
conflicts.  Readiness check can not compensate for user errors.  Therefore, it 
is a good idea to document readiness check would not compensate for this type 
of user errors to set expectations correctly.

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch, YARN-8060.5.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-8160) Yarn Service Upgrade: Support upgrade of docker containers

2018-04-13 Thread Chandni Singh (JIRA)

Chandni Singh created YARN-8160:
---

 Summary: Yarn Service Upgrade: Support upgrade of docker 
containers 
 Key: YARN-8160
 URL: https://issues.apache.org/jira/browse/YARN-8160
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Chandni Singh
Assignee: Chandni Singh


Ability to upgrade dockerized  yarn native services.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7996) Allow user supplied Docker client configurations with YARN native services

2018-04-13 Thread Shane Kumpf (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shane Kumpf updated YARN-7996:
--
Attachment: YARN-7996.005.patch

> Allow user supplied Docker client configurations with YARN native services
> --
>
> Key: YARN-7996
> URL: https://issues.apache.org/jira/browse/YARN-7996
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Attachments: YARN-7996.001.patch, YARN-7996.002.patch, 
> YARN-7996.003.patch, YARN-7996.004.patch, YARN-7996.005.patch
>
>
> YARN-5428 added support to distributed shell for supplying a Docker client 
> configuration at application submission time. The auth tokens within the 
> client configuration are then used to pull images from private Docker 
> repositories/registries. Add the same support to the YARN Native Services 
> framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437957#comment-16437957
 ] 

Gour Saha edited comment on YARN-8060 at 4/13/18 9:27 PM:
--

[~billie.rinaldi] thank you for the updated patch. +1 for patch 5 (assuming 
there is going to be nothing different in jenkins compared to patch 4, since 
only a log line got added). 

I have only 1 final very minor suggestion:
In Configurations.md, can you please add a header "Configurations" just like we 
have for all the other pages?


was (Author: gsaha):
[~billie.rinaldi] thank you for the updated patch. +1 for patch 5 (assuming 
there is going to be nothing different in jenkins compared to patch 4, since 
only a log line got added). I have only 1 final very minor suggestion:
In Configurations.md, can you please add a header "Configurations" just like we 
have for all the other pages?

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch, YARN-8060.5.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437957#comment-16437957
 ] 

Gour Saha commented on YARN-8060:
-

[~billie.rinaldi] thank you for the updated patch. +1 for patch 5 (assuming 
there is going to be nothing different in jenkins compared to patch 4, since 
only a log line got added). I have only 1 final very minor suggestion:
In Configurations.md, can you please add a header "Configurations" just like we 
have for all the other pages?

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch, YARN-8060.5.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437956#comment-16437956
 ] 

genericqa commented on YARN-8060:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 19s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
33s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 125 unchanged - 3 fixed = 125 total (was 128) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 34s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  8m 
45s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
16s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 50s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8060 |
| JIRA Patch URL |

[jira] [Commented] (YARN-7996) Allow user supplied Docker client configurations with YARN native services

2018-04-13 Thread Shane Kumpf (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437955#comment-16437955
 ] 

Shane Kumpf commented on YARN-7996:
---

Thanks for the review [~jlowe] and [~billie.rinaldi]. Attaching a patch that 
addresses the comments.
{quote}It's a little odd for validateDockerClientConfiguration to go through 
the motions to build a full URI but then limit that URI check to a particular 
filesystem. Is that intentional, or should it be calling Path#getFileSystem 
instead of assuming it should use fs.getFileSystem()?
{quote}
Thanks for calling this out. I was intending to validate that the user supplied 
a URI and that the URI exists. I see now that the URI check really isn't 
necessary and that reusing that existing fs object did limit usage to a single 
filesystem. I reworked that part to use {{Path#getFileSystem}} as suggested.

> Allow user supplied Docker client configurations with YARN native services
> --
>
> Key: YARN-7996
> URL: https://issues.apache.org/jira/browse/YARN-7996
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Attachments: YARN-7996.001.patch, YARN-7996.002.patch, 
> YARN-7996.003.patch, YARN-7996.004.patch
>
>
> YARN-5428 added support to distributed shell for supplying a Docker client 
> configuration at application submission time. The auth tokens within the 
> client configuration are then used to pull images from private Docker 
> repositories/registries. Add the same support to the YARN Native Services 
> framework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs

2018-04-13 Thread Kanwaljeet Sachdev (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437953#comment-16437953
 ] 

Kanwaljeet Sachdev commented on YARN-7088:
--

Thanks [~haibochen] for the comments. Here are my responses
{quote}We are generating the timestamp when RMAppImpl is notified of the 
launch. But the correct timestamp should come from 
RMAppAttemptEventType.LAUNCHED that is generated by AMLauncher. If the event 
dispatch thread is falling back, there could be a large gap between when 
RMAppImpl is notified and when the attempt is actually launched.
{quote}
I have addressed the above by adding a new constructor to update the timeStamp 
in the data members of _RMAppEvent and RMAppAttemptEvent_ classes.

 
{quote}I see two transitions are added to RMAPPImpl state machine, namely 
ACCEPTED --> ACCEPTED and RUNNING --> RUNNING upon the new 
RMAppEventType.ATTEMPT_LAUNCHED
{quote}
I have addressed this by removing the unwanted transition handling state change 
of RUNING --> RUNNING

 
{quote}Some minor comments: there is an unused variable you added in 
AMLaunchedTransition; the log in AppRunningOnAppAttemptLaunchTransition  is too 
generic. Can we add application id, attempt id and the timestamp?

The change in 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/server/application_history_server.proto
 is unnecessary as you have decided not to change for ATS (all versions). 
{quote}
Done as suggested

 
{quote}The schedulingWaitTime in AppInfo is misleading if one application has 
multiple attempts. Because the launch time is updated everytime a new attempt 
is launched, we can rename it to the launch time of the latest attempt. If 
there are mutiple attempts, the running time of previous attempts would be 
counted as schedulingwait time, which is not correct.
{quote}
I update the launchTime only if it is non zero so I guess we are fine here?

 

Four tests failed in the last run. Three are from 
*TestMRTimelineEventHandling*. I verified that they fail even without my 
changes and if I increase the timeout of the below snippet from 30 seconds, 
they pass, so my guess is they are flaky and not related to my changes.

 
{code:java}
public static RunningJob runJobSucceed(JobConf conf, Path inDir, Path outDir)
   throws IOException {
  conf.setJobName("test-job-succeed");
  conf.setMapperClass(IdentityMapper.class);
  conf.setReducerClass(IdentityReducer.class);
  
  RunningJob job = UtilsForTests.runJob(conf, inDir, outDir);
  long sleepCount = 0;
  while (!job.isComplete()) {
try {
  if (sleepCount > 300) { // 30 seconds
throw new IOException("Job didn't finish in 30 seconds");
  }
  Thread.sleep(100);
  sleepCount++;
} catch (InterruptedException e) {
  break;
}
  }
{code}
The other test 
*org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithContainerResourceChange*
 shows the same trace as reported by [~rchiang] in 
https://issues.apache.org/jira/browse/YARN-6272

 

 

> Fix application start time and add submit time to UIs
> -
>
> Key: YARN-7088
> URL: https://issues.apache.org/jira/browse/YARN-7088
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha4
>Reporter: Abdullah Yousufi
>Assignee: Kanwaljeet Sachdev
>Priority: Major
> Attachments: YARN-7088.001.patch, YARN-7088.002.patch, 
> YARN-7088.003.patch, YARN-7088.004.patch, YARN-7088.005.patch, 
> YARN-7088.006.patch, YARN-7088.007.patch, YARN-7088.008.patch, 
> YARN-7088.009.patch, YARN-7088.010.patch, YARN-7088.011.patch, 
> YARN-7088.012.patch, YARN-7088.013.patch, YARN-7088.014.patch, 
> YARN-7088.015.patch
>
>
> Currently, the start time in the old and new UI actually shows the app 
> submission time. There should actually be two different fields; one for the 
> app's submission and one for its start, as well as the elapsed pending time 
> between the two.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8108) RM metrics rest API throws GSSException in kerberized environment

2018-04-13 Thread Vinod Kumar Vavilapalli (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinod Kumar Vavilapalli updated YARN-8108:
--
Fix Version/s: (was: 3.0.0)

[~kbadani], please leave the fix-version empty. In Hadoop projects, we set it 
only at commit time. Tx.

> RM metrics rest API throws GSSException in kerberized environment
> -
>
> Key: YARN-8108
> URL: https://issues.apache.org/jira/browse/YARN-8108
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Kshitij Badani
>Priority: Major
>
> Test is trying to pull up metrics data from SHS after kiniting as 'test_user'
> It is throwing GSSException as follows
> {code:java}
> b2b460b80713|RUNNING: curl --silent -k -X GET -D 
> /hwqe/hadoopqe/artifacts/tmp-94845 --negotiate -u : 
> http://rm_host:8088/proxy/application_1518674952153_0070/metrics/json2018-02-15
>  07:15:48,757|INFO|MainThread|machine.py:194 - 
> run()||GUID=fc5a3266-28f8-4eed-bae2-b2b460b80713|Exit Code: 0
> 2018-02-15 07:15:48,758|INFO|MainThread|spark.py:1757 - 
> getMetricsJsonData()|metrics:
> 
> 
> 
> Error 403 GSSException: Failure unspecified at GSS-API level 
> (Mechanism level: Request is a replay (34))
> 
> HTTP ERROR 403
> Problem accessing /proxy/application_1518674952153_0070/metrics/json. 
> Reason:
>  GSSException: Failure unspecified at GSS-API level (Mechanism level: 
> Request is a replay (34))
> 
> 
> {code}
> Rootcausing : proxyserver on RM can't be supported for Kerberos enabled 
> cluster because AuthenticationFilter is applied twice in Hadoop code (once in 
> httpServer2 for RM, and another instance from AmFilterInitializer for proxy 
> server). This will require code changes to hadoop-yarn-server-web-proxy 
> project



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7189) Container-executor doesn't remove Docker containers that error out early

2018-04-13 Thread Eric Badger (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437908#comment-16437908
 ] 

Eric Badger commented on YARN-7189:
---

Sorry for the patch overload. Found a few nits right after I put up the patch. 
Patch 004 fixes those.

> Container-executor doesn't remove Docker containers that error out early
> 
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch, 
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch, 
> YARN-7189-branch-3.0.003.patch, YARN-7189-branch-3.0.004.patch
>
>
> Once the docker run command is executed, the docker container is created 
> unless the return code is 125 meaning that the run command itself failed 
> (https://docs.docker.com/engine/reference/run/#exit-status). Any error that 
> happens after the docker run needs to remove the container during cleanup.
> {noformat:title=container-executor.c:launch_docker_container_as_user}
>   snprintf(docker_command_with_binary, command_size, "%s %s", docker_binary, 
> docker_command);
>   fprintf(LOGFILE, "Launching docker container...\n");
>   FILE* start_docker = popen(docker_command_with_binary, "r");
> {noformat}
> This is fixed by YARN-5366, which changes how we remove containers. However, 
> that was committed into 3.1.0. 2.8, 2.9, and 3.0 are all affected



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7189) Container-executor doesn't remove Docker containers that error out early

2018-04-13 Thread Eric Badger (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-7189:
--
Attachment: YARN-7189-branch-3.0.004.patch

> Container-executor doesn't remove Docker containers that error out early
> 
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch, 
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch, 
> YARN-7189-branch-3.0.003.patch, YARN-7189-branch-3.0.004.patch
>
>
> Once the docker run command is executed, the docker container is created 
> unless the return code is 125 meaning that the run command itself failed 
> (https://docs.docker.com/engine/reference/run/#exit-status). Any error that 
> happens after the docker run needs to remove the container during cleanup.
> {noformat:title=container-executor.c:launch_docker_container_as_user}
>   snprintf(docker_command_with_binary, command_size, "%s %s", docker_binary, 
> docker_command);
>   fprintf(LOGFILE, "Launching docker container...\n");
>   FILE* start_docker = popen(docker_command_with_binary, "r");
> {noformat}
> This is fixed by YARN-5366, which changes how we remove containers. However, 
> that was committed into 3.1.0. 2.8, 2.9, and 3.0 are all affected



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8142) yarn service application stops when AM is killed with SIGTERM

2018-04-13 Thread Billie Rinaldi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437868#comment-16437868
 ] 

Billie Rinaldi commented on YARN-8142:
--

Thanks for the review and commit, [~eyang]! I plan to cherry-pick this to 
branch-3.1 if no one objects.

> yarn service application stops when AM is killed with SIGTERM
> -
>
> Key: YARN-8142
> URL: https://issues.apache.org/jira/browse/YARN-8142
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-native-services
>Reporter: Yesha Vora
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8142.1.patch
>
>
> Steps:
> 1) Launch sleeper job ( non-docker yarn service)
> {code}
> RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn app -launch 
> fault-test-am-sleeper 
> /usr/hdp/current/hadoop-yarn-client/yarn-service-examples/sleeper/sleeper.json
> WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of 
> YARN_LOG_DIR.
> WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of 
> YARN_LOGFILE.
> WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of 
> YARN_PID_DIR.
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 18/04/06 22:24:24 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> 18/04/06 22:24:24 INFO client.AHSProxy: Connecting to Application History 
> server at xxx:10200
> 18/04/06 22:24:24 INFO client.AHSProxy: Connecting to Application History 
> server at xxx:10200
> 18/04/06 22:24:24 INFO client.ApiServiceClient: Loading service definition 
> from local FS: 
> /usr/hdp/current/hadoop-yarn-client/yarn-service-examples/sleeper/sleeper.json
> 18/04/06 22:24:26 INFO util.log: Logging initialized @3631ms
> 18/04/06 22:24:37 INFO client.ApiServiceClient: Application ID: 
> application_1522887500374_0010
> Exit Code: 0{code}
> 2) Wait for sleeper component to be up
> 3) Kill AM process PID
>  
> Expected behavior:
> New attempt of AM will be started. The pre-existing container will keep 
> running
>  
> Actual behavior:
> Application finishes with State : FINISHED and Final-State : ENDED
> New attempt was never launched
> Note: 
> when the AM gets a SIGTERM and gracefully shuts itself down. It is shutting 
> the entire app down instead of letting it continue to run for another attempt
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7189) Container-executor doesn't remove Docker containers that error out early

2018-04-13 Thread Eric Badger (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437850#comment-16437850
 ] 

Eric Badger commented on YARN-7189:
---

Thanks for the review, [~jlowe]! Fixed up the patch to use the suggestions that 
you mentioned above. Again, tested it manually and it works in both the case 
where the removal works initially and when it fails initially and has to retry.

> Container-executor doesn't remove Docker containers that error out early
> 
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch, 
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch, 
> YARN-7189-branch-3.0.003.patch
>
>
> Once the docker run command is executed, the docker container is created 
> unless the return code is 125 meaning that the run command itself failed 
> (https://docs.docker.com/engine/reference/run/#exit-status). Any error that 
> happens after the docker run needs to remove the container during cleanup.
> {noformat:title=container-executor.c:launch_docker_container_as_user}
>   snprintf(docker_command_with_binary, command_size, "%s %s", docker_binary, 
> docker_command);
>   fprintf(LOGFILE, "Launching docker container...\n");
>   FILE* start_docker = popen(docker_command_with_binary, "r");
> {noformat}
> This is fixed by YARN-5366, which changes how we remove containers. However, 
> that was committed into 3.1.0. 2.8, 2.9, and 3.0 are all affected



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7189) Container-executor doesn't remove Docker containers that error out early

2018-04-13 Thread Eric Badger (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-7189:
--
Attachment: YARN-7189-branch-3.0.003.patch

> Container-executor doesn't remove Docker containers that error out early
> 
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch, 
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch, 
> YARN-7189-branch-3.0.003.patch
>
>
> Once the docker run command is executed, the docker container is created 
> unless the return code is 125 meaning that the run command itself failed 
> (https://docs.docker.com/engine/reference/run/#exit-status). Any error that 
> happens after the docker run needs to remove the container during cleanup.
> {noformat:title=container-executor.c:launch_docker_container_as_user}
>   snprintf(docker_command_with_binary, command_size, "%s %s", docker_binary, 
> docker_command);
>   fprintf(LOGFILE, "Launching docker container...\n");
>   FILE* start_docker = popen(docker_command_with_binary, "r");
> {noformat}
> This is fixed by YARN-5366, which changes how we remove containers. However, 
> that was committed into 3.1.0. 2.8, 2.9, and 3.0 are all affected



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7939) Yarn Service Upgrade: add support to upgrade a component instance

2018-04-13 Thread Chandni Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437773#comment-16437773
 ] 

Chandni Singh edited comment on YARN-7939 at 4/13/18 8:08 PM:
--

[~eyang]
{quote}
2018-04-13 17:23:08,321 ERROR client.ApiServiceClient: Failed to start service 
abc, because it already exists.
{quote}
I think the upgrade of the service was not initiated due to some error and the 
service is NOT in "UPGRADING" state which is why you cannot restart it when it 
is already running.

I will look into making hbase application work


was (Author: csingh):
[~eyang]
{quote}
2018-04-13 17:23:08,321 ERROR client.ApiServiceClient: Failed to start service 
abc, because it already exists.
{quote}
I think the upgrade of the service was not initiated due to some error and the 
service is NOT in "UPGRADING" state which is why you cannot restart it when it 
already running.

> Yarn Service Upgrade: add support to upgrade a component instance 
> --
>
> Key: YARN-7939
> URL: https://issues.apache.org/jira/browse/YARN-7939
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-7939.001.patch, YARN-7939.002.patch, 
> YARN-7939.003.patch, YARN-7939.004.patch, YARN-7939.005.patch, 
> YARN-7939.006.patch, YARN-7939.007.patch
>
>
> Yarn core supports in-place upgrade of containers. A yarn service can 
> leverage that to provide in-place upgrade of component instances. Please see 
> YARN-7512 for details.
> Will add support to upgrade a single component instance first and then 
> iteratively add other APIs and features.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8142) yarn service application stops when AM is killed with SIGTERM

2018-04-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437825#comment-16437825
 ] 

Hudson commented on YARN-8142:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13995 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13995/])
YARN-8142.  Improve SIGTERM handling for YARN Service Application (eyang: rev 
9031a76d447f0c5eaa392144fd17c5b9812e1b20)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ClientAMService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java


> yarn service application stops when AM is killed with SIGTERM
> -
>
> Key: YARN-8142
> URL: https://issues.apache.org/jira/browse/YARN-8142
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-native-services
>Reporter: Yesha Vora
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8142.1.patch
>
>
> Steps:
> 1) Launch sleeper job ( non-docker yarn service)
> {code}
> RUNNING: /usr/hdp/current/hadoop-yarn-client/bin/yarn app -launch 
> fault-test-am-sleeper 
> /usr/hdp/current/hadoop-yarn-client/yarn-service-examples/sleeper/sleeper.json
> WARNING: YARN_LOG_DIR has been replaced by HADOOP_LOG_DIR. Using value of 
> YARN_LOG_DIR.
> WARNING: YARN_LOGFILE has been replaced by HADOOP_LOGFILE. Using value of 
> YARN_LOGFILE.
> WARNING: YARN_PID_DIR has been replaced by HADOOP_PID_DIR. Using value of 
> YARN_PID_DIR.
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 18/04/06 22:24:24 WARN util.NativeCodeLoader: Unable to load native-hadoop 
> library for your platform... using builtin-java classes where applicable
> 18/04/06 22:24:24 INFO client.AHSProxy: Connecting to Application History 
> server at xxx:10200
> 18/04/06 22:24:24 INFO client.AHSProxy: Connecting to Application History 
> server at xxx:10200
> 18/04/06 22:24:24 INFO client.ApiServiceClient: Loading service definition 
> from local FS: 
> /usr/hdp/current/hadoop-yarn-client/yarn-service-examples/sleeper/sleeper.json
> 18/04/06 22:24:26 INFO util.log: Logging initialized @3631ms
> 18/04/06 22:24:37 INFO client.ApiServiceClient: Application ID: 
> application_1522887500374_0010
> Exit Code: 0{code}
> 2) Wait for sleeper component to be up
> 3) Kill AM process PID
>  
> Expected behavior:
> New attempt of AM will be started. The pre-existing container will keep 
> running
>  
> Actual behavior:
> Application finishes with State : FINISHED and Final-State : ENDED
> New attempt was never launched
> Note: 
> when the AM gets a SIGTERM and gracefully shuts itself down. It is shutting 
> the entire app down instead of letting it continue to run for another attempt
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.

2018-04-13 Thread Eric Payne (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437778#comment-16437778
 ] 

Eric Payne commented on YARN-4781:
--

[~sunilg], sorry for the churn on here, but I think we need to get this right. 
I think we need to take a step back and decide what exactly the FairOrdering 
Preemption policy should do.

Your previous suggestion that it should consider pending as well as used when 
that feature is enabled caused me to begin thinking that we wanted FairOrdering 
Preemption to order apps in the same exact way as the FairOrderingApp policy 
does for assigning containers. However, the {{FairComparator}} does not compare 
which users are most underserved. Another part of the code does that. So, it's 
not enough for FairOrdering Preemption to just call into the {{FairComparator}}

I suggest that rather than trying to re-use the {{FairComparator}}, 
FairOrdering Preemption should mimic the behavior of the Capacity Scheduler 
{{FairOrderingPolicy}}. I know that would require a bit of a maintenance risk 
since if the behavior of one changes, the other must follow. However, I think 
that would be the simplest option at this point.

So, I think that the {{TAFairOrderingComparator}} should do what it does in my 
latest patch with 2 additions:
- if {{fair.enable-size-based-weight}} is enabled for the queue, consider 
pending resources as well
- if 2 apps are from the same user, the younger one is defined to be more 
underserved (and therefore will be preempted after the older one)
 

> Support intra-queue preemption for fairness ordering policy.
> 
>
> Key: YARN-4781
> URL: https://issues.apache.org/jira/browse/YARN-4781
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Attachments: YARN-4781.001.patch, YARN-4781.002.patch, 
> YARN-4781.003.patch
>
>
> We introduced fairness queue policy since YARN-3319, which will let large 
> applications make progresses and not starve small applications. However, if a 
> large application takes the queue’s resources, and containers of the large 
> app has long lifespan, small applications could still wait for resources for 
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to 
> preempt resources of queue with fairness policy enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7939) Yarn Service Upgrade: add support to upgrade a component instance

2018-04-13 Thread Chandni Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437773#comment-16437773
 ] 

Chandni Singh commented on YARN-7939:
-

[~eyang]
{quote}
2018-04-13 17:23:08,321 ERROR client.ApiServiceClient: Failed to start service 
abc, because it already exists.
{quote}
I think the upgrade of the service was not initiated due to some error and the 
service is NOT in "UPGRADING" state which is why you cannot restart it when it 
already running.

> Yarn Service Upgrade: add support to upgrade a component instance 
> --
>
> Key: YARN-7939
> URL: https://issues.apache.org/jira/browse/YARN-7939
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Attachments: YARN-7939.001.patch, YARN-7939.002.patch, 
> YARN-7939.003.patch, YARN-7939.004.patch, YARN-7939.005.patch, 
> YARN-7939.006.patch, YARN-7939.007.patch
>
>
> Yarn core supports in-place upgrade of containers. A yarn service can 
> leverage that to provide in-place upgrade of component instances. Please see 
> YARN-7512 for details.
> Will add support to upgrade a single component instance first and then 
> iteratively add other APIs and features.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437769#comment-16437769
 ] 

genericqa commented on YARN-8060:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 29s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
21s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 125 unchanged - 3 fixed = 125 total (was 128) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 14s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
1s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  7m 
19s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
35s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 88m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8060 |
| JIRA Patch URL |

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Billie Rinaldi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437767#comment-16437767
 ] 

Billie Rinaldi commented on YARN-8060:
--

[~yeshavora] just mentioned she'd like it to be logged when the readiness check 
succeeds, too, so I added a line for that in patch 5.

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch, YARN-8060.5.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Billie Rinaldi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-8060:
-
Attachment: YARN-8060.5.patch

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch, YARN-8060.5.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7939) Yarn Service Upgrade: add support to upgrade a component instance

2018-04-13 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437696#comment-16437696
 ] 

Eric Yang commented on YARN-7939:
-

[~csingh] With tarball application, patch 007 shows this diagnostics info when 
trigger instances to upgrade:
{code}
AM Container for appattempt_1523637126365_0003_01 exited with exitCode: -100
Failing this attempt.Diagnostics: Container released on a *lost* nodeFor more 
detailed output, check the application tracking page: 
http://eyang-2.openstacklocal:8088/cluster/app/application_1523637126365_0003 
Then click on links to logs of each attempt.
{code}

AM log file shows:
{code}
2018-04-13 17:04:53,019 [AMRM Callback Handler Thread] WARN  
service.ServiceScheduler - Nodes updated info: 
eyang-3.openstacklocal:49921, state = UNHEALTHY, healthDiagnostics = Linux 
Container Executor reached unrecoverable exception

2018-04-13 17:04:53,022 [AMRM Callback Handler Thread] WARN  
service.ServiceScheduler - Container container_1523637126365_0003_01_02 
Completed. No component instance exists. exitStatus=-100. diagnostics=Container 
released on a *lost* node 
2018-04-13 17:04:53,041 [main] INFO  registry.YarnRegistryViewForProviders - 
Resolving path 
/users/hbase/services/yarn-service/abc/components/ctr-1523637126365-0003-01-02
2018-04-13 17:04:53,043 [main] INFO  service.ServiceScheduler - Handling 
container_1523637126365_0003_01_03 from previous attempt
2018-04-13 17:04:53,045 [main] INFO  component.Component - [COMPONENT ping]: 
Recovered container_1523637126365_0003_01_03 for component instance ping-1 
on host eyang-4.openstacklocal:49039, num pending component instances reduced 
to 1 
2018-04-13 17:04:53,047 [main] INFO  service.ServiceScheduler - Triggering 
initial evaluation of component ping
2018-04-13 17:04:53,047 [main] INFO  component.Component - [INIT COMPONENT 
ping]: 2 instances.
2018-04-13 17:04:53,047 [main] INFO  component.Component - [COMPONENT ping] 
Requesting for 0 container(s)
2018-04-13 17:04:53,048 [main] INFO  component.Component - [COMPONENT ping] 
Transitioned from INIT to FLEXING on FLEX event.
2018-04-13 17:04:53,049 [pool-5-thread-1] INFO  service.ServiceScheduler - 
Registered service under /users/hbase/services/yarn-service/abc; absolute path 
/registry/users/hbase/services/yarn-service/abc
2018-04-13 17:04:53,062 [Component  dispatcher] INFO  
instance.ComponentInstance - [COMPINSTANCE ping-1 : 
container_1523637126365_0003_01_03] Transitioned from INIT to STARTED on 
START event
2018-04-13 17:04:53,127 [pool-5-thread-1] INFO  instance.ComponentInstance - 
[COMPINSTANCE ping-1 : container_1523637126365_0003_01_03] IP = 
[172.26.111.21], host = eyang-4.openstacklocal, cancel container status 
retriever
2018-04-13 17:05:23,055 [pool-7-thread-1] INFO  instance.ComponentInstance - 
[COMPINSTANCE ping-1 : container_1523637126365_0003_01_03] Transitioned 
from STARTED to READY on BECOME_READY event
2018-04-13 17:06:53,055 [pool-5-thread-3] INFO  service.ServiceScheduler - 
[COMPINSTANCE ping-0], wait on container container_1523637126365_0003_01_02 
expired
2018-04-13 17:06:53,056 [pool-5-thread-3] INFO  
registry.YarnRegistryViewForProviders - [COMPINSTANCE ping-0]: Deleting 
registry path 
/users/hbase/services/yarn-service/abc/components/ctr-1523637126365-0003-01-02
2018-04-13 17:06:53,088 [pool-5-thread-3] INFO  component.Component - 
[COMPONENT ping] Requesting for 1 container(s)
2018-04-13 17:06:53,099 [pool-5-thread-3] INFO  component.Component - 
[COMPONENT ping] Submitting container request : Capability[]Priority[0]AllocationRequestId[0]ExecutionTypeRequest[{Execution 
Type: GUARANTEED, Enforce Execution Type: false}]Resource Profile[null]
2018-04-13 17:06:54,507 [AMRM Callback Handler Thread] INFO  
service.ServiceScheduler - 1 containers allocated. 
2018-04-13 17:06:54,508 [Component  dispatcher] INFO  component.Component - 
[COMPONENT ping]: container_1523637126365_0003_02_02 allocated, num pending 
component instances reduced to 0
2018-04-13 17:06:54,508 [Component  dispatcher] INFO  component.Component - 
[COMPONENT ping]: Assigned container_1523637126365_0003_02_02 to component 
instance ping-0 and launch on host eyang-4.openstacklocal:49039 
2018-04-13 17:06:54,509 [AMRM Callback Handler Thread] INFO  
service.ServiceScheduler - [COMPONENT ping]: remove 1 outstanding container 
requests for allocateId 0
2018-04-13 17:06:54,546 [pool-6-thread-1] INFO  provider.ProviderUtils - 
Component instance conf dir already exists: 
hdfs://eyang-1.openstacklocal:9000/user/hbase/.yarn/services/abc/components/v1/ping/ping-0
2018-04-13 17:06:54,551 [pool-6-thread-1] INFO  
containerlaunch.ContainerLaunchService - launching container 
container_1523637126365_0003_02_02
2018-04-13 17:06:54,561 
[org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #0] INFO  
impl.NMClientAsyncImpl

[jira] [Updated] (YARN-6538) Inter Queue preemption is not happening when DRF is configured

2018-04-13 Thread Wangda Tan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-6538:
-
Issue Type: Sub-task  (was: Bug)
Parent: YARN-8159

> Inter Queue preemption is not happening when DRF is configured
> --
>
> Key: YARN-6538
> URL: https://issues.apache.org/jira/browse/YARN-6538
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
>
> Cluster capacity of . Here memory is more and vcores 
> are less. If applications have more demand, vcores might be exhausted. 
> Inter queue preemption ideally has to be kicked in once vcores is over 
> utilized. However preemption is not happening.
> Analysis:
> In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
> {code}
> // assign all cluster resources until no more demand, or no resources are
> // left
> while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
> unassigned, Resources.none())) {
> {code}
>  will loop even when vcores are 0 (because memory is still +ve). Hence we are 
> having more vcores in idealAssigned which cause no-preemption cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8020) when DRF is used, preemption does not trigger due to incorrect idealAssigned

2018-04-13 Thread Wangda Tan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-8020:
-
Issue Type: Sub-task  (was: Bug)
Parent: YARN-8159

> when DRF is used, preemption does not trigger due to incorrect idealAssigned
> 
>
> Key: YARN-8020
> URL: https://issues.apache.org/jira/browse/YARN-8020
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: kyungwan nam
>Priority: Major
>
> I’ve met that Inter Queue Preemption does not work.
> It happens when DRF is used and submitting application with a large number of 
> vcores.
> IMHO, idealAssigned can be set incorrectly by following code.
> {code}
> // This function "accepts" all the resources it can (pending) and return
> // the unused ones
> Resource offer(Resource avail, ResourceCalculator rc,
> Resource clusterResource, boolean considersReservedResource) {
>   Resource absMaxCapIdealAssignedDelta = Resources.componentwiseMax(
>   Resources.subtract(getMax(), idealAssigned),
>   Resource.newInstance(0, 0));
>   // accepted = min{avail,
>   //   max - assigned,
>   //   current + pending - assigned,
>   //   # Make sure a queue will not get more than max of its
>   //   # used/guaranteed, this is to make sure preemption won't
>   //   # happen if all active queues are beyond their guaranteed
>   //   # This is for leaf queue only.
>   //   max(guaranteed, used) - assigned}
>   // remain = avail - accepted
>   Resource accepted = Resources.min(rc, clusterResource,
>   absMaxCapIdealAssignedDelta,
>   Resources.min(rc, clusterResource, avail, Resources
>   /*
>* When we're using FifoPreemptionSelector (considerReservedResource
>* = false).
>*
>* We should deduct reserved resource from pending to avoid 
> excessive
>* preemption:
>*
>* For example, if an under-utilized queue has used = reserved = 20.
>* Preemption policy will try to preempt 20 containers (which is not
>* satisfied) from different hosts.
>*
>* In FifoPreemptionSelector, there's no guarantee that preempted
>* resource can be used by pending request, so policy will preempt
>* resources repeatly.
>*/
>   .subtract(Resources.add(getUsed(),
>   (considersReservedResource ? pending : pendingDeductReserved)),
>   idealAssigned)));
> {code}
> let’s say,
> * cluster resource : 
> * idealAssigned(assigned): 
> * avail: 
> * current: 
> * pending: 
> current + pending - assigned: 
> min ( avail, (current + pending - assigned) ) : 
> accepted: 
> as a result, idealAssigned will be , which does not 
> trigger preemption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Billie Rinaldi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437624#comment-16437624
 ] 

Billie Rinaldi commented on YARN-8060:
--

Uploaded patch 4 which enhances the documentation to include which 
configuration properties are allowed to be configured in which places. This was 
a useful exercise, because I discovered that the container recovery timeout was 
at the system level when it should have been at the service level, and the 
container retry properties were at the service level when they should have been 
at the component level. I fixed this in the patch. I also realized that the 
readiness check configuration properties were not documented except in the 
javadocs, so I added a section on those.

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8091) Revisit checkUserAccessToQueue RM REST API

2018-04-13 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated YARN-8091:

Fix Version/s: 3.1.1

Cherry-picked to branch-3.1.

> Revisit checkUserAccessToQueue RM REST API
> --
>
> Key: YARN-8091
> URL: https://issues.apache.org/jira/browse/YARN-8091
> Project: Hadoop YARN
>  Issue Type: Task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Critical
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8091.001.patch
>
>
> As offline suggested by [~sershe]. Currently design of the 
> checkUserAccessToQueue mixed config-related issues (like user doesn't access 
> to the URL) and user-facing output (like requested user is not permitted to 
> access the queue) in the same code.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8028) Support authorizeUserAccessToQueue in RMWebServices

2018-04-13 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated YARN-8028:

Fix Version/s: 3.1.1

Cherry-picked to branch-3.1.

> Support authorizeUserAccessToQueue in RMWebServices
> ---
>
> Key: YARN-8028
> URL: https://issues.apache.org/jira/browse/YARN-8028
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8028.001.patch, YARN-8028.002.patch
>
>
> Currently we have {{QueueUserACLInfo}} in ApplicationClient, we should 
> support similar API in REST API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8048) Support auto-spawning of admin configured services during bootstrap of rm/apiserver

2018-04-13 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated YARN-8048:

Fix Version/s: 3.1.1

Cherry-picked to branch-3.1.

> Support auto-spawning of admin configured services during bootstrap of 
> rm/apiserver
> ---
>
> Key: YARN-8048
> URL: https://issues.apache.org/jira/browse/YARN-8048
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8048.001.patch, YARN-8048.002.patch, 
> YARN-8048.003.patch, YARN-8048.004.patch, YARN-8048.005.patch, 
> YARN-8048.006.patch
>
>
> Goal is to support auto-spawning of admin configured services during 
> bootstrap of resourcemanager/apiserver. 
> *Requirement:* Some of the  services might required to be consumed by yarn 
> itself ex: Hbase for atsv2. Instead of depending on user installed HBase or 
> sometimes user may not required to install HBase at all, in such conditions 
> running HBase app on YARN will help for ATSv2.
> Before YARN cluster is started, admin configure these services spec and place 
> it in common location in HDFS. At the time of RM/apiServer bootstrap, these 
> services will be submitted.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8040) [UI2] New YARN UI webapp does not respect current pathname for REST api

2018-04-13 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated YARN-8040:

Fix Version/s: 3.1.1

Cherry-picked to branch-3.1.

> [UI2] New YARN UI webapp does not respect current pathname for REST api
> ---
>
> Key: YARN-8040
> URL: https://issues.apache.org/jira/browse/YARN-8040
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8040.001.patch
>
>
> When ui2 is accessed behind proxy like knox/nginx, trailing path name should 
> not be skipped. However trim of "ui2" if its there.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-1151) Ability to configure auxiliary services from HDFS-based JAR files

2018-04-13 Thread Arpit Agarwal (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal updated YARN-1151:

Fix Version/s: 3.1.1

Cherry-picked to branch-3.1.

> Ability to configure auxiliary services from HDFS-based JAR files
> -
>
> Key: YARN-1151
> URL: https://issues.apache.org/jira/browse/YARN-1151
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 2.1.0-beta, 2.9.0
>Reporter: john lilley
>Assignee: Xuan Gong
>Priority: Major
>  Labels: auxiliary-service, yarn
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-1151.1.patch, YARN-1151.2.patch, YARN-1151.3.patch, 
> YARN-1151.4.patch, YARN-1151.5.patch, YARN-1151.6.patch, 
> YARN-1151.branch-2.poc.2.patch, YARN-1151.branch-2.poc.3.patch, 
> YARN-1151.branch-2.poc.patch, [YARN-1151] [Design] Configure auxiliary 
> services from HDFS-based JAR files.pdf
>
>
> I would like to install an auxiliary service in Hadoop YARN without actually 
> installing files/services on every node in the system.  Discussions on the 
> user@ list indicate that this is not easily done.  The reason we want an 
> auxiliary service is that our application has some persistent-data components 
> that are not appropriate for HDFS.  In fact, they are somewhat analogous to 
> the mapper output of MapReduce's shuffle, which is what led me to 
> auxiliary-services in the first place.  It would be much easier if we could 
> just place our service's JARs in HDFS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Billie Rinaldi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-8060:
-
Attachment: YARN-8060.4.patch

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, 
> YARN-8060.4.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Reopened] (YARN-7810) TestDockerContainerRuntime test failures due to UID lookup of a non-existent user

2018-04-13 Thread Wei-Chiu Chuang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang reopened YARN-7810:
---

Sorry I have to reopen this issue. I reverted the commit on branch-2.

> TestDockerContainerRuntime test failures due to UID lookup of a non-existent 
> user
> -
>
> Key: YARN-7810
> URL: https://issues.apache.org/jira/browse/YARN-7810
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
> Attachments: YARN-7810-branch-2.001.patch, 
> YARN-7810-branch-3.0.001.patch, YARN-7810.001.patch, YARN-7810.002.patch
>
>
> YARN-7782 enabled the Docker runtime feature to remap the username to uid:gid 
> form for launching Docker containers. The feature does an {{id -u}} and {{id 
> -G}} to get the UID and GIDs. This fails with the test user, as that user 
> doesn't actually exist on the host.
> {code:java}
> [ERROR] 
> testContainerLaunchWithCustomNetworks(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime)
>   Time elapsed: 0.411 s  <<< ERROR!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
>  
> ExitCodeException exitCode=1: id: 'run_as_user': no such user
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.getUserIdInfo(DockerLinuxContainerRuntime.java:711)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:757)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks(TestDockerContainerRuntime.java:599){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7810) TestDockerContainerRuntime test failures due to UID lookup of a non-existent user

2018-04-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437600#comment-16437600
 ] 

Wei-Chiu Chuang edited comment on YARN-7810 at 4/13/18 5:14 PM:


Hi [~eyang] [~ebadger], the branch-2 is broken because of this commit (for 
Java7)

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile 
(default-testCompile) on project hadoop-yarn-server-nodemanager: Compilation 
failure: Compilation failure:
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[344,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[512,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[580,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[625,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[799,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[918,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String
[ERROR] 
/Users/weichiu/sandbox/upstream/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java:[994,48]
 cannot find symbol
[ERROR] symbol:   method join(java.lang.String,java.lang.String[])
[ERROR] location: class java.lang.String


was (Author: jojochuang):
Hi [~eyang] [~ebadger], the branch-2 is broken because of this commit (for 
Java7)

> TestDockerContainerRuntime test failures due to UID lookup of a non-existent 
> user
> -
>
> Key: YARN-7810
> URL: https://issues.apache.org/jira/browse/YARN-7810
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
> Attachments: YARN-7810-branch-2.001.patch, 
> YARN-7810-branch-3.0.001.patch, YARN-7810.001.patch, YARN-7810.002.patch
>
>
> YARN-7782 enabled the Docker runtime feature to remap the username to uid:gid 
> form for launching Docker containers. The feature does an {{id -u}} and {{id 
> -G}} to get the UID and GIDs. This fails with the test user, as that user 
> doesn't actually exist on the host.
> {code:java}
> [ERROR] 
> testContainerLaunchWithCustomNetworks(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime)
>   Time elapsed: 0.411 s  <<< ERROR!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
>  
> ExitCodeException exitCode=1: id: 'run_as_user': no such user
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.getUserIdInfo(DockerLinuxContainerRuntime.java:711)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:757)
>   at 
>

[jira] [Commented] (YARN-7810) TestDockerContainerRuntime test failures due to UID lookup of a non-existent user

2018-04-13 Thread Wei-Chiu Chuang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437600#comment-16437600
 ] 

Wei-Chiu Chuang commented on YARN-7810:
---

Hi [~eyang] [~ebadger], the branch-2 is broken because of this commit (for 
Java7)

> TestDockerContainerRuntime test failures due to UID lookup of a non-existent 
> user
> -
>
> Key: YARN-7810
> URL: https://issues.apache.org/jira/browse/YARN-7810
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
> Attachments: YARN-7810-branch-2.001.patch, 
> YARN-7810-branch-3.0.001.patch, YARN-7810.001.patch, YARN-7810.002.patch
>
>
> YARN-7782 enabled the Docker runtime feature to remap the username to uid:gid 
> form for launching Docker containers. The feature does an {{id -u}} and {{id 
> -G}} to get the UID and GIDs. This fails with the test user, as that user 
> doesn't actually exist on the host.
> {code:java}
> [ERROR] 
> testContainerLaunchWithCustomNetworks(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime)
>   Time elapsed: 0.411 s  <<< ERROR!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
>  
> ExitCodeException exitCode=1: id: 'run_as_user': no such user
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.getUserIdInfo(DockerLinuxContainerRuntime.java:711)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:757)
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks(TestDockerContainerRuntime.java:599){code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5094) some YARN container events have timestamp of -1

2018-04-13 Thread Haibo Chen (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-5094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haibo Chen updated YARN-5094:
-
Component/s: (was: timelineserver)

> some YARN container events have timestamp of -1
> ---
>
> Key: YARN-5094
> URL: https://issues.apache.org/jira/browse/YARN-5094
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Sangjin Lee
>Assignee: Haibo Chen
>Priority: Critical
>  Labels: YARN-5355
> Fix For: 3.1.0, 2.9.1, 3.0.1
>
> Attachments: YARN-5094-YARN-2928.001.patch, YARN-5094.00.patch, 
> YARN-5094.02.patch
>
>
> Some events in the YARN container entities have timestamp of -1. The 
> RM-generated container events have proper timestamps. It appears that it's 
> the NM-generated events that have -1: YARN_CONTAINER_CREATED, 
> YARN_CONTAINER_FINISHED, YARN_NM_CONTAINER_LOCALIZATION_FINISHED, 
> YARN_NM_CONTAINER_LOCALIZATION_STARTED.
> In the YARN container page,
> {noformat}
> {
> id: "YARN_CONTAINER_CREATED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_CONTAINER_FINISHED",
> timestamp: -1,
> info: {
> YARN_CONTAINER_EXIT_STATUS: 0,
> YARN_CONTAINER_STATE: "RUNNING",
> YARN_CONTAINER_DIAGNOSTICS_INFO: ""
> }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_FINISHED",
> timestamp: -1,
> info: { }
> },
> {
> id: "YARN_NM_CONTAINER_LOCALIZATION_STARTED",
> timestamp: -1,
> info: { }
> }
> {noformat}
> I think the data itself is OK, but the values are not being populated in the 
> REST output?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8071) Add ability to specify nodemanager environment variables individually

2018-04-13 Thread Jim Brennan (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437537#comment-16437537
 ] 

Jim Brennan commented on YARN-8071:
---

[~jlowe], I believe this patch is ready for review.

 

> Add ability to specify nodemanager environment variables individually
> -
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-8071.001.patch, YARN-8071.002.patch
>
>
> YARN-6830 describes a problem where environment variables that contain commas 
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to 
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>  
> -Dmapreduce.map.env.MODE=bar
>  -Dmapreduce.map.env.IMAGE_NAME=foo
>  -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what 
> is actually being defined.
> {quote}
> The mapreduce properties were dealt with in [MAPREDUCE-7069].  This Jira will 
> address the YARN properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8071) Add ability to specify nodemanager environment variables individually

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437502#comment-16437502
 ] 

genericqa commented on YARN-8071:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 
 2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 14s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m  8s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
3s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 
24s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 93m 13s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8071 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12918956/YARN-8071.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  xml  findbugs  checkstyle  |
| uname | Linux fc0067816ea6 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0725953 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
|  Test

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437478#comment-16437478
 ] 

Gour Saha commented on YARN-8060:
-

My bad on the log-line in HttpProbe.java. The code snippet I provided above was 
actually removed and it has been correctly added with comp-instance name in 
DefaultProbe.java as below -
{code}
  status.fail(this, new IOException(
  instance.getCompInstanceName() + ": IP is not available yet"));
{code}

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Billie Rinaldi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437474#comment-16437474
 ] 

Billie Rinaldi commented on YARN-8060:
--

bq. I think we should add a new section in Configurations.md listing all these 
system overridable configurations.
We're on the same page. The new patch I have started working on breaks the 
configurations down into system, service, and component level configurations.

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Gour Saha (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437469#comment-16437469
 ] 

Gour Saha commented on YARN-8060:
-

{quote} let's open a separate ticket to discuss allowing multiple checks
{quote}
Totally makes sense
{quote} ...it is allowing you to adjust the system-wide default instead of 
having all systems stuck with the default of 10...
{quote}
This is a great point and I did not think along this line. This is a nice way 
to allow and consume cluster level defaults than every service having to 
specify them if they don't like dev-chosen defaults. I think we should add a 
new section in {{Configurations.md}} listing all these system overridable 
configurations.

> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8108) RM metrics rest API throws GSSException in kerberized environment

2018-04-13 Thread Eric Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437464#comment-16437464
 ] 

Eric Yang commented on YARN-8108:
-

The possible solution is to make proxyserver configurable to disable 
AmFilterInitializer when proxyserver is running as embedded web application on 
resource manager.
[~daryn] Any thoughts?

> RM metrics rest API throws GSSException in kerberized environment
> -
>
> Key: YARN-8108
> URL: https://issues.apache.org/jira/browse/YARN-8108
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Kshitij Badani
>Priority: Major
> Fix For: 3.0.0
>
>
> Test is trying to pull up metrics data from SHS after kiniting as 'test_user'
> It is throwing GSSException as follows
> {code:java}
> b2b460b80713|RUNNING: curl --silent -k -X GET -D 
> /hwqe/hadoopqe/artifacts/tmp-94845 --negotiate -u : 
> http://rm_host:8088/proxy/application_1518674952153_0070/metrics/json2018-02-15
>  07:15:48,757|INFO|MainThread|machine.py:194 - 
> run()||GUID=fc5a3266-28f8-4eed-bae2-b2b460b80713|Exit Code: 0
> 2018-02-15 07:15:48,758|INFO|MainThread|spark.py:1757 - 
> getMetricsJsonData()|metrics:
> 
> 
> 
> Error 403 GSSException: Failure unspecified at GSS-API level 
> (Mechanism level: Request is a replay (34))
> 
> HTTP ERROR 403
> Problem accessing /proxy/application_1518674952153_0070/metrics/json. 
> Reason:
>  GSSException: Failure unspecified at GSS-API level (Mechanism level: 
> Request is a replay (34))
> 
> 
> {code}
> Rootcausing : proxyserver on RM can't be supported for Kerberos enabled 
> cluster because AuthenticationFilter is applied twice in Hadoop code (once in 
> httpServer2 for RM, and another instance from AmFilterInitializer for proxy 
> server). This will require code changes to hadoop-yarn-server-web-proxy 
> project



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6495) check docker container's exit code when writing to cgroup task files

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437456#comment-16437456
 ] 

genericqa commented on YARN-6495:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m 23s{color} 
| {color:red} YARN-6495 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6495 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12918965/YARN-6495.002.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/20340/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> check docker container's exit code when writing to cgroup task files
> 
>
> Key: YARN-6495
> URL: https://issues.apache.org/jira/browse/YARN-6495
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
>Priority: Major
> Attachments: YARN-6495.001.patch, YARN-6495.002.patch
>
>
> If I execute simple command like date on docker container, the application 
> failed to complete successfully.
> for example, 
> {code}
> $ yarn  jar 
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
>  -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env 
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker -shell_command "date" -jar 
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
>  -num_containers 1 -timeout 360
> …
> 17/04/12 00:16:40 INFO distributedshell.Client: Application did finished 
> unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring 
> loop
> 17/04/12 00:16:40 ERROR distributedshell.Client: Application failed to 
> complete successfully
> {code}
> The error log is like below.
> {code}
> ...
> Failed to write pid to file 
> /cgroup_parent/cpu/hadoop-yarn/container_/tasks - No such process
> ...
> {code}
> When writing pid to cgroup tasks, container-executor doesn’t check docker 
> container’s status.
> If the container finished very quickly, we can’t write pid to cgroup tasks, 
> and it is not problem.
> So container-executor needs to check docker container’s exit code during 
> writing pid to cgroup tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8104) Add API to fetch node to attribute mapping

2018-04-13 Thread Bibin A Chundatt (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437412#comment-16437412
 ] 

Bibin A Chundatt commented on YARN-8104:


As discussed offline updating patch with minor variable name change in proto

> Add API to fetch node to attribute mapping
> --
>
> Key: YARN-8104
> URL: https://issues.apache.org/jira/browse/YARN-8104
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8104-YARN-3409.001.patch, 
> YARN-8104-YARN-3409.002.patch, YARN-8104-YARN-3409.003.patch, 
> YARN-8104-YARN-3409.004.patch, YARN-8104-YARN-3409.005.patch, 
> YARN-8104-YARN-3409.006.patch
>
>
> Add node/host to attribute mapping in yarn client API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8104) Add API to fetch node to attribute mapping

2018-04-13 Thread Bibin A Chundatt (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-8104:
---
Attachment: YARN-8104-YARN-3409.006.patch

> Add API to fetch node to attribute mapping
> --
>
> Key: YARN-8104
> URL: https://issues.apache.org/jira/browse/YARN-8104
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-8104-YARN-3409.001.patch, 
> YARN-8104-YARN-3409.002.patch, YARN-8104-YARN-3409.003.patch, 
> YARN-8104-YARN-3409.004.patch, YARN-8104-YARN-3409.005.patch, 
> YARN-8104-YARN-3409.006.patch
>
>
> Add node/host to attribute mapping in yarn client API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6495) check docker container's exit code when writing to cgroup task files

2018-04-13 Thread Jaeboo Jeong (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437403#comment-16437403
 ] 

Jaeboo Jeong commented on YARN-6495:


I separated docker container's exit code from cgroup failed exit code. And I 
uploaded the patch file.

> check docker container's exit code when writing to cgroup task files
> 
>
> Key: YARN-6495
> URL: https://issues.apache.org/jira/browse/YARN-6495
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
>Priority: Major
> Attachments: YARN-6495.001.patch, YARN-6495.002.patch
>
>
> If I execute simple command like date on docker container, the application 
> failed to complete successfully.
> for example, 
> {code}
> $ yarn  jar 
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
>  -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env 
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker -shell_command "date" -jar 
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
>  -num_containers 1 -timeout 360
> …
> 17/04/12 00:16:40 INFO distributedshell.Client: Application did finished 
> unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring 
> loop
> 17/04/12 00:16:40 ERROR distributedshell.Client: Application failed to 
> complete successfully
> {code}
> The error log is like below.
> {code}
> ...
> Failed to write pid to file 
> /cgroup_parent/cpu/hadoop-yarn/container_/tasks - No such process
> ...
> {code}
> When writing pid to cgroup tasks, container-executor doesn’t check docker 
> container’s status.
> If the container finished very quickly, we can’t write pid to cgroup tasks, 
> and it is not problem.
> So container-executor needs to check docker container’s exit code during 
> writing pid to cgroup tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6495) check docker container's exit code when writing to cgroup task files

2018-04-13 Thread Jaeboo Jeong (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaeboo Jeong updated YARN-6495:
---
Attachment: YARN-6495.002.patch

> check docker container's exit code when writing to cgroup task files
> 
>
> Key: YARN-6495
> URL: https://issues.apache.org/jira/browse/YARN-6495
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
>Priority: Major
> Attachments: YARN-6495.001.patch, YARN-6495.002.patch
>
>
> If I execute simple command like date on docker container, the application 
> failed to complete successfully.
> for example, 
> {code}
> $ yarn  jar 
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
>  -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env 
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker -shell_command "date" -jar 
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
>  -num_containers 1 -timeout 360
> …
> 17/04/12 00:16:40 INFO distributedshell.Client: Application did finished 
> unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring 
> loop
> 17/04/12 00:16:40 ERROR distributedshell.Client: Application failed to 
> complete successfully
> {code}
> The error log is like below.
> {code}
> ...
> Failed to write pid to file 
> /cgroup_parent/cpu/hadoop-yarn/container_/tasks - No such process
> ...
> {code}
> When writing pid to cgroup tasks, container-executor doesn’t check docker 
> container’s status.
> If the container finished very quickly, we can’t write pid to cgroup tasks, 
> and it is not problem.
> So container-executor needs to check docker container’s exit code during 
> writing pid to cgroup tasks.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7799) YARN Service dependency follow up work

2018-04-13 Thread Billie Rinaldi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437353#comment-16437353
 ] 

Billie Rinaldi commented on YARN-7799:
--

I think this is a critical issue that we should try to get done for 3.1.1.

> YARN Service dependency follow up work
> --
>
> Key: YARN-7799
> URL: https://issues.apache.org/jira/browse/YARN-7799
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client, resourcemanager
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Critical
>
> As per [~jianhe] these are some followup items that make sense to do after 
> YARN-7766. Quoting Jian's comment below -
> Currently, if user doesn't supply location when run yarn app 
> -enableFastLaunch, the jars will be put under this location
> {code}
> hdfs:///yarn-services//service-dep.tar.gz
> {code}
> Since API server is embedded in RM, should RM look for this location too if 
> "yarn.service.framework.path" is not specified ?
> And if "yarn.service.framework.path" is not specified and still the file 
> doesn't exist at above default location, I think RM can try to upload the 
> jars to above default location instead, currently RM is uploading the jars to 
> the location defined by below code. This folder is per app and also 
> inconsistent with CLI location.
> {code}
>   protected Path addJarResource(String serviceName,
>   Map localResources)
>   throws IOException, SliderException {
> Path libPath = fs.buildClusterDirPath(serviceName);
> {code}
> By doing this, the next time a submission request comes, RM doesn't need to 
> upload the jars again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7799) YARN Service dependency follow up work

2018-04-13 Thread Billie Rinaldi (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Billie Rinaldi updated YARN-7799:
-
Target Version/s: 3.2.0, 3.1.1  (was: 3.2.0)

> YARN Service dependency follow up work
> --
>
> Key: YARN-7799
> URL: https://issues.apache.org/jira/browse/YARN-7799
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: client, resourcemanager
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Critical
>
> As per [~jianhe] these are some followup items that make sense to do after 
> YARN-7766. Quoting Jian's comment below -
> Currently, if user doesn't supply location when run yarn app 
> -enableFastLaunch, the jars will be put under this location
> {code}
> hdfs:///yarn-services//service-dep.tar.gz
> {code}
> Since API server is embedded in RM, should RM look for this location too if 
> "yarn.service.framework.path" is not specified ?
> And if "yarn.service.framework.path" is not specified and still the file 
> doesn't exist at above default location, I think RM can try to upload the 
> jars to above default location instead, currently RM is uploading the jars to 
> the location defined by below code. This folder is per app and also 
> inconsistent with CLI location.
> {code}
>   protected Path addJarResource(String serviceName,
>   Map localResources)
>   throws IOException, SliderException {
> Path libPath = fs.buildClusterDirPath(serviceName);
> {code}
> By doing this, the next time a submission request comes, RM doesn't need to 
> upload the jars again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8060) Create default readiness check for service components

2018-04-13 Thread Billie Rinaldi (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437342#comment-16437342
 ] 

Billie Rinaldi commented on YARN-8060:
--

Thanks for the reviews, [~shaneku...@gmail.com] and [~gsaha]!

bq. do you think it would make sense to change readiness_check to 
readiness_checks and support multiple checks
This idea has crossed my mind before, but I don't think it is necessary for the 
fixes made in this patch. I like how each of the basic checks has optional DNS 
checking. It's kind of difficult to design a feature without having a specific 
use case for it, but since the idea keeps coming up, let's open a separate 
ticket to discuss allowing multiple checks.

bq. Should we honor CONTAINER_FAILURE_THRESHOLD specified at the system level?
It's not exactly that the service/component-level property is being specified 
at the system level, it's more that it is allowing you to adjust the 
system-wide default instead of having all systems stuck with the default of 10. 
That's how I view the YarnServiceConf getters; they use hardcoded defaults 
unless different defaults are specified in the yarn-site.xml file.

bq. The component instance name is not getting logged in the fail message.
That is handled in the log line, LOG.info("Readiness check failed for {}: {}", 
instance.getCompInstanceName(), status). Although, now that you mention it, I 
should be using our convention of "[COMPONENT {}]" at the beginning of the log 
line text.

Thanks again. I'll make the rest of the changes and upload a new patch.


> Create default readiness check for service components
> -
>
> Key: YARN-8060
> URL: https://issues.apache.org/jira/browse/YARN-8060
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
>Priority: Major
> Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch
>
>
> It is currently possible for a component instance to have READY status before 
> the AM retrieves an IP for the container. We should make sure the IP has been 
> retrieved before marking the instance as READY.
> This default probe could also have an option to check for a DNS entry for the 
> instance's hostname if a DNS address is provided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8071) Add ability to specify nodemanager environment variables individually

2018-04-13 Thread Jim Brennan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Brennan updated YARN-8071:
--
Attachment: YARN-8071.002.patch

> Add ability to specify nodemanager environment variables individually
> -
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-8071.001.patch, YARN-8071.002.patch
>
>
> YARN-6830 describes a problem where environment variables that contain commas 
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to 
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>  
> -Dmapreduce.map.env.MODE=bar
>  -Dmapreduce.map.env.IMAGE_NAME=foo
>  -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what 
> is actually being defined.
> {quote}
> The mapreduce properties were dealt with in [MAPREDUCE-7069].  This Jira will 
> address the YARN properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8071) Add ability to specify nodemanager environment variables individually

2018-04-13 Thread Jim Brennan (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Brennan updated YARN-8071:
--
Description: 
YARN-6830 describes a problem where environment variables that contain commas 
cannot be specified via {{-Dmapreduce.map.env}}.

For example:

{{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}

will set {{MOUNTS}} to {{/tmp/foo}}

In that Jira, [~aw] suggested that we change the API to provide a way to 
specify environment variables individually, the same way that Spark does.
{quote}Rather than fight with a regex why not redefine the API instead?

 

-Dmapreduce.map.env.MODE=bar
 -Dmapreduce.map.env.IMAGE_NAME=foo
 -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar

...

e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar

This greatly simplifies the input validation needed and makes it clear what is 
actually being defined.
{quote}

The mapreduce properties were dealt with in [MAPREDUCE-7069].  This Jira will 
address the YARN properties.

  was:
YARN-6830 describes a problem where environment variables that contain commas 
cannot be specified via {{-Dmapreduce.map.env}}.

For example:

{{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}

will set {{MOUNTS}} to {{/tmp/foo}}

In that Jira, [~aw] suggested that we change the API to provide a way to 
specify environment variables individually, the same way that Spark does.
{quote}Rather than fight with a regex why not redefine the API instead?

 

-Dmapreduce.map.env.MODE=bar
 -Dmapreduce.map.env.IMAGE_NAME=foo
 -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar

...

e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar

This greatly simplifies the input validation needed and makes it clear what is 
actually being defined.
{quote}


> Add ability to specify nodemanager environment variables individually
> -
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-8071.001.patch
>
>
> YARN-6830 describes a problem where environment variables that contain commas 
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to 
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>  
> -Dmapreduce.map.env.MODE=bar
>  -Dmapreduce.map.env.IMAGE_NAME=foo
>  -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what 
> is actually being defined.
> {quote}
> The mapreduce properties were dealt with in [MAPREDUCE-7069].  This Jira will 
> address the YARN properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437308#comment-16437308
 ] 

genericqa commented on YARN-8015:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 15s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 2 new + 37 unchanged - 4 fixed = 39 total (was 41) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 15s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m 
29s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m  1s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8015 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12918932/YARN-8015.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 60b6071a61a2 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0725953 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/20337/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/20337/testReport/ |
| Max. process+thread count | 821 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U:

[jira] [Comment Edited] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437185#comment-16437185
 ] 

Weiwei Yang edited comment on YARN-8015 at 4/13/18 12:10 PM:
-

Hi [~leftnoteasy] could you please help to review?

FYI, I think we need a Jira to revisit how to support node-partition in PC, I 
can see some problems/limitation in current approach (which only supports 
affinity to a partition).


was (Author: cheersyang):
Hi [~leftnoteasy] could you please help to review?

> Support inter-app placement constraints in AppPlacementAllocator
> 
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch, YARN-8015.002.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437185#comment-16437185
 ] 

Weiwei Yang edited comment on YARN-8015 at 4/13/18 12:10 PM:
-

Hi [~leftnoteasy] 

Could you please help to review?

FYI, I think we need a Jira to revisit how to support node-partition in PC, I 
can see some problems/limitation in current approach (which only supports 
affinity to a partition).


was (Author: cheersyang):
Hi [~leftnoteasy] could you please help to review?

FYI, I think we need a Jira to revisit how to support node-partition in PC, I 
can see some problems/limitation in current approach (which only supports 
affinity to a partition).

> Support inter-app placement constraints in AppPlacementAllocator
> 
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch, YARN-8015.002.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8153) Guaranteed containers always stay in SCHEDULED on NM after restart

2018-04-13 Thread Yang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437186#comment-16437186
 ] 

Yang Wang commented on YARN-8153:
-

[~cheersyang] Thanks for your commit.

> Guaranteed containers always stay in SCHEDULED on NM after restart
> --
>
> Key: YARN-8153
> URL: https://issues.apache.org/jira/browse/YARN-8153
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8153.001.patch, YARN-8153.002.patch
>
>
> When nm recovery is enabled, after NM restart, some containers always stay in 
> SCHEDULED because of no sufficient resources.
> The root cause is that utilizationTracker.addContainerResources has been 
> called twice when restart. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437185#comment-16437185
 ] 

Weiwei Yang commented on YARN-8015:
---

Hi [~leftnoteasy] could you please help to review?

> Support inter-app placement constraints in AppPlacementAllocator
> 
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch, YARN-8015.002.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437174#comment-16437174
 ] 

Weiwei Yang commented on YARN-8015:
---

Fix checkstyle issues as much as possible..

> Support inter-app placement constraints in AppPlacementAllocator
> 
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch, YARN-8015.002.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-8015:
--
Attachment: YARN-8015.002.patch

> Support inter-app placement constraints in AppPlacementAllocator
> 
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch, YARN-8015.002.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437164#comment-16437164
 ] 

genericqa commented on YARN-8015:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 27s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 10 new + 37 unchanged - 4 fixed = 47 total (was 41) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 10s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m 
41s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8015 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12918903/YARN-8015.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 2bf51551576b 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 
21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1a407bc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/20336/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/20336/testReport/ |
| Max. process+thread count | 809 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U:

[jira] [Commented] (YARN-8154) Fix missing titles in PlacementConstraints document

2018-04-13 Thread Akira Ajisaka (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437026#comment-16437026
 ] 

Akira Ajisaka commented on YARN-8154:
-

Thanks [~cheersyang]!

> Fix missing titles in PlacementConstraints document
> ---
>
> Key: YARN-8154
> URL: https://issues.apache.org/jira/browse/YARN-8154
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Affects Versions: 3.1.0
>Reporter: Akira Ajisaka
>Assignee: Weiwei Yang
>Priority: Minor
>  Labels: newbie
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8154.001.patch, YARN-8154.002.patch, 
> fix_missing_H3_titles.jpg
>
>
> Any section title starts with '### ' is missing.
> http://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8015) Support inter-app placement constraints in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-8015:
--
Summary: Support inter-app placement constraints in AppPlacementAllocator  
(was: Support allocation tag namespaces in AppPlacementAllocator)

> Support inter-app placement constraints in AppPlacementAllocator
> 
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-8015) Support allocation tag namespaces in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-8015:
--
Attachment: YARN-8015.001.patch

> Support allocation tag namespaces in AppPlacementAllocator
> --
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8015.001.patch
>
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8015) Support allocation tag namespaces in AppPlacementAllocator

2018-04-13 Thread Weiwei Yang (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436989#comment-16436989
 ] 

Weiwei Yang commented on YARN-8015:
---

Uploaded v1 patch, changes outline:

Added some UT in {{TestSchedulingRequestContainerAllocation}} to cover:
 * Inter-app constraints by namespace, including all/not-self/app-id/app-tag
 * Inter-app composite constraints, specify constraints like AND(P1, P2)
 * Inter-app constraints with multiple source targets

Simplified validations in {{SingleConstraintAppPlacementAllocator}}, removed 
unnecessary checks including
 * Only allow single constraint
 * Only support node scope
 * Only support anti-affinity

Remove unnecessary validations in {{TestSingleConstraintAppPlacementAllocator}} 
accordingly

> Support allocation tag namespaces in AppPlacementAllocator
> --
>
> Key: YARN-8015
> URL: https://issues.apache.org/jira/browse/YARN-8015
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
>
> AppPlacementAllocator currently only supports intra-app anti-affinity 
> placement constraints, once YARN-8002 and YARN-8013 are resolved, it needs to 
> support inter-app constraints too. Also, this may require some refactoring on 
> the existing code logic. Use this JIRA to track.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7598) Document how to use classpath isolation for aux-services in YARN

2018-04-13 Thread Junping Du (JIRA)


 [ 
https://issues.apache.org/jira/browse/YARN-7598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated YARN-7598:
-
Target Version/s: 3.2.0, 3.1.1, 2.9.2, 3.0.3  (was: 3.2.0)

> Document how to use classpath isolation for aux-services in YARN
> 
>
> Key: YARN-7598
> URL: https://issues.apache.org/jira/browse/YARN-7598
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: YARN-7598.2.patch, YARN-7598.3.patch, YARN-7598.4.patch, 
> YARN-7598.trunk.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7939) Yarn Service Upgrade: add support to upgrade a component instance

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436950#comment-16436950
 ] 

genericqa commented on YARN-7939:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 12 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 
 4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 13s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 14s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 33 new + 401 unchanged - 2 fixed = 434 total (was 403) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 33s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 28m  
2s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  6m 
52s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
31s{color} | {color:green} hadoop-yarn-services-api in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
27s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 12s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-7939 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12918885/YARN-7939.007.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux 5e4cd71f89cb 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build

[jira] [Commented] (YARN-8156) Increase the default value of yarn.timeline-service.app-collector.linger-period.ms

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436931#comment-16436931
 ] 

genericqa commented on YARN-8156:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m  
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 56s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
 9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 39s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  3m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
12s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  
3s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
34s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 51s{color} | 
{color:black} {color} |
\\
\\
||

[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs

2018-04-13 Thread genericqa (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436919#comment-16436919
 ] 

genericqa commented on YARN-7088:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 15 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 30m 
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  3m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  5m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
19m 41s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  7m 
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  3m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 26m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 26m 
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 26m 
58s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
3m 25s{color} | {color:orange} root: The patch generated 2 new + 1118 unchanged 
- 2 fixed = 1120 total (was 1120) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  5m 
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 20s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  8m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  3m 
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
48s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
17s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
19s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
54s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the 
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m 
27s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 28s{color} 
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}141m 10s{color} 
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
53s{color} | {color:green} The patch does not generate ASF License warnings.

[jira] [Commented] (YARN-8153) Guaranteed containers always stay in SCHEDULED on NM after restart

2018-04-13 Thread Hudson (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436893#comment-16436893
 ] 

Hudson commented on YARN-8153:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13990 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13990/])
YARN-8153. Guaranteed containers always stay in SCHEDULED on NM after (wwei: 
rev 226bedc0239ba23f3ca0c40dac6aab3777d3ada6)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java


> Guaranteed containers always stay in SCHEDULED on NM after restart
> --
>
> Key: YARN-8153
> URL: https://issues.apache.org/jira/browse/YARN-8153
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8153.001.patch, YARN-8153.002.patch
>
>
> When nm recovery is enabled, after NM restart, some containers always stay in 
> SCHEDULED because of no sufficient resources.
> The root cause is that utilizationTracker.addContainerResources has been 
> called twice when restart. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8156) Increase the default value of yarn.timeline-service.app-collector.linger-period.ms

2018-04-13 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436890#comment-16436890
 ] 

Rohith Sharma K S commented on YARN-8156:
-

+lgtm, pending jenkins

> Increase the default value of 
> yarn.timeline-service.app-collector.linger-period.ms
> --
>
> Key: YARN-8156
> URL: https://issues.apache.org/jira/browse/YARN-8156
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-8156.001.patch, YARN-8156.002.patch
>
>
> The default value of *yarn.timeline-service.app-collector.linger-period.ms* 
> is 1 sec which is very low. In production cluster, we observe that most of 
> the entity information is not published because of 1sec linger period. 
> I would suggest to increase the default value to either 10sec or 30sec or 60 
> sec. As far as I have seen, 10sec should be sufficient since node heartbeat 
> period is 3 sec by default, all NM gets request to stop the application. 
>  cc:/ [~haibo.chen] [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-8156) Increase the default value of yarn.timeline-service.app-collector.linger-period.ms

2018-04-13 Thread Rohith Sharma K S (JIRA)


[ 
https://issues.apache.org/jira/browse/YARN-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436887#comment-16436887
 ] 

Rohith Sharma K S commented on YARN-8156:
-

{quote}Does the default setting increase to 60 second in the patch not help?
{quote}
I think its better to set for 1sec because test case might wait for 60 seconds 
and gets timeout or takes a  minute to complete. Setting to1sec will make test 
to pass quickly.  

> Increase the default value of 
> yarn.timeline-service.app-collector.linger-period.ms
> --
>
> Key: YARN-8156
> URL: https://issues.apache.org/jira/browse/YARN-8156
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Charan Hebri
>Priority: Major
> Attachments: YARN-8156.001.patch, YARN-8156.002.patch
>
>
> The default value of *yarn.timeline-service.app-collector.linger-period.ms* 
> is 1 sec which is very low. In production cluster, we observe that most of 
> the entity information is not published because of 1sec linger period. 
> I would suggest to increase the default value to either 10sec or 30sec or 60 
> sec. As far as I have seen, 10sec should be sufficient since node heartbeat 
> period is 3 sec by default, all NM gets request to stop the application. 
>  cc:/ [~haibo.chen] [~vrushalic]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

1 2 >

100 matches

Mail list logo