subject:"\[jira\] \[Commented\] \(YARN\-9860\) Enable service mode for Docker containers on YARN"

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-10 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949191#comment-16949191
 ] 

Prabhu Joseph commented on YARN-9860:
-

Thanks [~eyang] and [~shaneku...@gmail.com].

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: Screen Shot 2019-10-09 at 11.27.19 AM.png, 
> YARN-9860-001.patch, YARN-9860-002.patch, YARN-9860-003.patch, 
> YARN-9860-004.patch, YARN-9860-005.patch, YARN-9860-006.patch, 
> YARN-9860-007.patch, YARN-9860-008.patch, YARN-9860-009.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-10 Thread Hudson (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949005#comment-16949005
 ] 

Hudson commented on YARN-9860:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17521 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/17521/])
YARN-9860. Enable service mode for Docker containers on YARN(eyang: 
rev 31e0122f4d4ddc4026470b45d2bf683ece137d44)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/YarnServiceConstants.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/tarball/TarballProviderService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/provider/TestProviderUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/SliderFileSystem.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ConfigFile.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRunCommand.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/ProviderUtils.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/CoreFileSystem.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h


> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Screen Shot 2019-10-09 at 11.27.19 AM.png, 
> YARN-9860-001.patch, YARN-9860-002.patch, YARN-9860-003.patch, 
> YARN-9860-004.patch, YARN-9860-005.patch, YARN-9860-006.patch, 
> YARN-9860-007.patch, YARN-9860-008.patch, YARN-9860-009.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#8

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-10 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948809#comment-16948809
 ] 

Eric Yang commented on YARN-9860:
-

[~Prabhu Joseph] Thank you for the patch.  

+1 Patch 009 looks good to me.  TestFederationInterceptor transient failure is 
not related to this issue.

Will commit today, if no objections.

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Screen Shot 2019-10-09 at 11.27.19 AM.png, 
> YARN-9860-001.patch, YARN-9860-002.patch, YARN-9860-003.patch, 
> YARN-9860-004.patch, YARN-9860-005.patch, YARN-9860-006.patch, 
> YARN-9860-007.patch, YARN-9860-008.patch, YARN-9860-009.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-10 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948422#comment-16948422
 ] 

Hadoop QA commented on YARN-9860:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
41s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 15s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
49s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  8m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
23s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 92 unchanged - 1 fixed = 92 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  5s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 17s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
56s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
42s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 45s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.amrmpr

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-10 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948312#comment-16948312
 ] 

Prabhu Joseph commented on YARN-9860:
-

[~eyang] Have updated container-executor.cfg config table and Application 
submission table in the Document in  [^YARN-9860-009.patch] . Thanks.




> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Screen Shot 2019-10-09 at 11.27.19 AM.png, 
> YARN-9860-001.patch, YARN-9860-002.patch, YARN-9860-003.patch, 
> YARN-9860-004.patch, YARN-9860-005.patch, YARN-9860-006.patch, 
> YARN-9860-007.patch, YARN-9860-008.patch, YARN-9860-009.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-09 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947430#comment-16947430
 ] 

Hadoop QA commented on YARN-9860:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
 8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 49s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 91 unchanged - 1 fixed = 91 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  0s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  
0s{color} | {color:blue} Skipped patched modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
12s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
55s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
23s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
40s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.3 Server=19.03.3

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-08 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947380#comment-16947380
 ] 

Prabhu Joseph commented on YARN-9860:
-

[~eyang] Have updated the DockerContainers.md with instructions for enabling 
service mode in [^YARN-9860-008.patch] . Thanks.

!Screen Shot 2019-10-09 at 11.27.19 AM.png|height=300!

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Screen Shot 2019-10-09 at 11.27.19 AM.png, 
> YARN-9860-001.patch, YARN-9860-002.patch, YARN-9860-003.patch, 
> YARN-9860-004.patch, YARN-9860-005.patch, YARN-9860-006.patch, 
> YARN-9860-007.patch, YARN-9860-008.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-08 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947379#comment-16947379
 ] 

Hadoop QA commented on YARN-9860:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  9s{color} 
| {color:red} YARN-9860 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9860 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/24939/console |
| Powered by | Apache Yetus 0.8.0   http://yetus.apache.org |


This message was automatically generated.



> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: Screen Shot 2019-10-09 at 11.27.19 AM.png, 
> YARN-9860-001.patch, YARN-9860-002.patch, YARN-9860-003.patch, 
> YARN-9860-004.patch, YARN-9860-005.patch, YARN-9860-006.patch, 
> YARN-9860-007.patch, YARN-9860-008.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-08 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947170#comment-16947170
 ] 

Eric Yang commented on YARN-9860:
-

[~Prabhu Joseph] Thanks for point out the user error.  The user principal name 
was misconfigured on my cluster to cause the error.  There is clear message 
about the error in application master log file.  I tested service mode=true and 
false.  Both cases are working as expected and keytab file localization is set 
with proper permission.  Please include instructions for enabling service mode 
in DockerContainers.md.  Thanks again.

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch, 
> YARN-9860-006.patch, YARN-9860-007.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-08 Thread Shane Kumpf (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946765#comment-16946765
 ] 

Shane Kumpf commented on YARN-9860:
---

{quote}Have changed the Keytab file visibility to PRIVATE and left others with 
default APPLICATION and can be overridden by user.\{quote}

I don't think we want to change existing behavior with this patch. If it was 
previously APPLICATION, I think it should stay APPLICATION. If it really should 
be PRIVATE, that should be a follow up. I expect moving those resources to 
private is the cause of the exception above.

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch, 
> YARN-9860-006.patch, YARN-9860-007.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-07 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946374#comment-16946374
 ] 

Eric Yang commented on YARN-9860:
-

[~Prabhu Joseph] Thank you for the patch 007.  I tested the patch on a kerberos 
enabled cluster, and containers fail to launch on the cluster.  Node manager 
error log shows:

{code}
2019-10-07 17:49:30,962 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService:
 Localizer failed for container_1570493636313_0004_02_01
java.io.IOException: Application application_1570493636313_0004 initialization 
failed (exitCode=-1) with output: null
at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:414)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:1263)
Caused by: 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationException:
 java.io.InterruptedIOException: java.lang.InterruptedException
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:185)
at 
org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.startLocalizer(LinuxContainerExecutor.java:405)
... 1 more
Caused by: java.io.InterruptedIOException: java.lang.InterruptedException
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1011)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperationExecutor.executePrivilegedOperation(PrivilegedOperationExecutor.java:154)
... 2 more
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1001)
... 5 more
{code}

I also checked the working directory.  There is no localization of keytab file 
or tokens.  Can you verify?

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch, 
> YARN-9860-006.patch, YARN-9860-007.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-05 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945157#comment-16945157
 ] 

Hadoop QA commented on YARN-9860:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
44s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
 3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
18m 36s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 10m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
41s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 91 unchanged - 1 fixed = 91 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 16s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m  
7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m  
1s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  1m 
 4s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 44s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12982285/YARN-9860-007.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux e14195d1a995 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 55c5436 |
| maven

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-05 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945138#comment-16945138
 ] 

Prabhu Joseph commented on YARN-9860:
-

[~eyang] 1. Have used the value of hadoop.tmp.dir for public resource directory.
2. The existing behavior by default sets APPLICATION visibility 
for all files to be localized including keytab.

* * CoreFileSystem#createAmResource*
{code}
 // Setting to most private option
-amResource.setVisibility(LocalResourceVisibility.APPLICATION);
+if (visibility == null) {
+  visibility = LocalResourceVisibility.APPLICATION;
+}
+amResource.setVisibility(visibility);
{code}

Have changed the Keytab file visibility to PRIVATE and left others with default 
APPLICATION and can be overridden by user.


> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch, 
> YARN-9860-006.patch, YARN-9860-007.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-04 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944878#comment-16944878
 ] 

Eric Yang commented on YARN-9860:
-

[~Prabhu Joseph]
{code}
+  String SERVICES_PUBLIC_DIRECTORY = "/tmp/hadoop-yarn/staging/";
+
{code}

I think this should map to hadoop.tmp.dir directory prefix for correctness.  I 
don't understand the resource localization changes in this patch.  It will set 
keytab file visibility to application, I don't think this is correct.  What is 
the purpose to set private resources to be application visibility?

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch, 
> YARN-9860-006.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-04 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944829#comment-16944829
 ] 

Hadoop QA commented on YARN-9860:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  1m 
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  9m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 40s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 91 unchanged - 1 fixed = 91 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  4s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
14s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 22s{color} 
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}127m 37s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.service.component.TestComponentDecommissionInstances |
|   | hadoop.yarn.service.TestServiceAM |
|   | hadoop.yarn.service.TestYarnNativeServices |
|   | hadoop.yarn.service.monitor.TestServiceMonitor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12982246/YARN-9860-006.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname |

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-04 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944743#comment-16944743
 ] 

Prabhu Joseph commented on YARN-9860:
-

Thanks [~eyang] and [~shaneku...@gmail.com] for reviewing. Have addressed the 
review comments in  [^YARN-9860-006.patch] .

1. Have refactored the docker-util.c with is_feature_enabled(). Tested with
Native Service Job, Service Mode enable and disable works fine.

2. Ragarding System.getenv("USER"), have verified that the USER environment
variable of container points to the actual user submitted the job irrespective 
of value 
of yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user. 
Containers
are run as per the config, but still the staging, log directories are of job 
user.

{code}
container-executor  

when yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user = 
nobody

/HADOOP/hadoop-3.3.0-SNAPSHOT/bin/container-executor nobody ambari-qa
{code}


> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch, 
> YARN-9860-006.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-04 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944678#comment-16944678
 ] 

Prabhu Joseph commented on YARN-9860:
-

Please ignore  [^YARN-9860-005.patch]  which has a bug in docker-util.c in the 
refactor code. WIll fix the same.

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch, 
> YARN-9860-003.patch, YARN-9860-004.patch, YARN-9860-005.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-04 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944671#comment-16944671
 ] 

Hadoop QA commented on YARN-9860:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
43s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 47s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated 
0 new + 92 unchanged - 1 fixed = 92 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  3s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
34s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 
14s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
46s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12982237/YARN-9860-005.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux dc6f6ba7b700 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 531cc93 |
| maven

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-04 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944417#comment-16944417
 ] 

Hadoop QA commented on YARN-9860:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 31s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
10s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 16s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 90 unchanged - 1 fixed = 91 total (was 91) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 55s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
38s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
56s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
47s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 22s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:1dde3efb91e |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12982192/YARN-9860-004.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux e383fde9def2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2478cba |
| m

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-03 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16944124#comment-16944124
 ] 

Hadoop QA commented on YARN-9860:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
15m 27s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
58s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 16s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 36 new + 91 unchanged - 1 fixed = 127 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 21s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
15s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
47s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 41s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.2 Server=19.03.2 Image:yetus/hadoop:0e026cb0cef |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12982163/YARN-9860-003.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux c9cf1d7e1526 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 9446686 |
|

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-02 Thread Shane Kumpf (Jira)

[
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942957#comment-16942957
]

Shane Kumpf commented on YARN-9860:
---

{quote}It looks like we are reverting to our old habit of using 0 for true. It
would be more consistent to use is_feature_enabled() method to determine if
service_mode is enabled, and reduce some code debris.
{quote}
Good points, I agree these comments should be addressed.
{quote}Container-executor already dup the container run output into stdout and
stderr log files with proper user permission for entrypoint mode because the
log files are initialized as user who runs the container executor rather than
the user in the container. It works for both secure and non-secure mode. I fail
to see the need to craft logging mechanism for the given reasoning for service
mode. Let me know if I missed something.
{quote}
Excellent, I can confirm this is working exactly how we'd want. I overlooked
this before. Seems logging isn't an issue after all. Thanks for pointing that
out!

I did retest the patch today and it is still working as expected.

With the patch applied in my dev VM, below is the ps and logs from the official
postgres image running under YARN with zero changes!

*ps:*
{code:java}
root@centos7-0:/# ps -ef
UIDPID PPID C STIME TTY TIME CMD
postgres 1 0 0 16:13 ?00:00:00 postgres
postgres53 1 0 16:13 ?00:00:00 postgres: checkpointer
postgres54 1 0 16:13 ?00:00:00 postgres: background writer
postgres55 1 0 16:13 ?00:00:00 postgres: walwriter
postgres56 1 0 16:13 ?00:00:00 postgres: autovacuum launcher
postgres57 1 0 16:13 ?00:00:00 postgres: stats collector
postgres58 1 0 16:13 ?00:00:00 postgres: logical replication
launcher
root59 0 4 16:14 pts/000:00:00 bash
root6459 0 16:14 pts/000:00:00 ps -ef
{code}

*Logs:*
{code:java}
[root@y7001 ~]# yarn logs -applicationId application_1570018164872_0005
-containerId container_1570018164872_0005_01_02
2019-10-02 16:26:40,269 INFO client.RMProxy: Connecting to ResourceManager at
y7001.yns.foo.com/192.168.70.211:9104
Container: container_1570018164872_0005_01_02 on y7001.yns.foo.com:9105
LogAggregationType: LOCAL
===
LogType:stdout.txt
LogLastModifiedTime:Wed Oct 02 16:13:31 + 2019
LogLength:2638
LogContents:
Launching docker container...
Docker run command: /usr/bin/docker run
--name=container_1570018164872_0005_01_02 --net=host -v
/tmp/hadoop-yarn/nm-local-dir/filecache/13/httpd-proxy.conf:/etc/httpd/conf.d/httpd-proxy.conf:ro
--cgroup-parent=/hadoop-yarn/container_1570018164872_0005_01_02
--cap-drop=ALL --cap-add=SYS_CHROOT --cap-add=MKNOD --cap-add=SETFCAP
--cap-add=SETPCAP --cap-add=FSETID --cap-add=CHOWN --cap-add=AUDIT_WRITE
--cap-add=SETGID --cap-add=NET_RAW --cap-add=FOWNER --cap-add=SETUID
--cap-add=DAC_OVERRIDE --cap-add=KILL --cap-add=NET_BIND_SERVICE
--hostname=centos7-0.skumpftest.hadoopuser.ynsdev --env-file
/tmp/hadoop-yarn/nm-local-dir/nmPrivate/application_1570018164872_0005/container_1570018164872_0005_01_02/docker.container_1570018164872_0005_01_028354800474788286290.env
library/postgres
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.UTF-8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default timezone ... Etc/UTC
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start2019-10-02 16:13:31.231 UTC [43] LOG: listening
on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2019-10-02 16:13:31.253 UTC [44] LOG: database system was shut down at
2019-10-02 16:13:30 UTC
2019-10-02 16:13:31.259 UTC [43] LOG: database system is ready to accept
connections
done
server started

/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

waiting for server to shut down...2019-10-02 16:13:31.322 UTC [43] LOG:
received fast shutdown request
.2019-10-02 16:13:31.325 UTC [43] LOG: aborting any active transactions
2019-10-02 16:13:31.329 UTC [43] LOG: background worker "logi

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-02 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942921#comment-16942921
 ] 

Eric Yang commented on YARN-9860:
-

{quote}That code does the opposite of what you stated, user is only passed when 
service mode is NOT enabled, which is what we want.{quote}

It looks like we are reverting to our old habit of using 0 for true.  It would 
be more consistent to use is_feature_enabled() method to determine if 
service_mode is enabled, and reduce some code debris.

{quote}You are spot on that this will be an issue. The challenge is that if we 
mount the read-write log dirs into the container and the container user isn't 
the user YARN expects, the writes could fail or YARN may be unable to clean up 
the logs. I talked with Craig on this a bit and he had some interesting 
thoughts on how we might handle it with fuse. For the sake of this patch, I 
didn't want to get bogged down in the details there, given this has enough 
going already. Could we address logging in a follow up? In the meantime, with 
debug delay enabled, doing a docker logs on the exited container will allow 
admins to take a look, since the output redirection typically done by YARN is 
dropped in Service Mode.{quote}

Container-executor already 
[dup|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c#L1982]
 the container run output into stdout and stderr log files with proper user 
permission for entrypoint mode because the log files are initialized as user 
who runs the container executor rather than the user in the container.  It 
works for both secure and non-secure mode.  I fail to see the need to craft 
logging mechanism for the given reasoning for service mode.  Let me know if I 
missed something.

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-02 Thread Eric Badger (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942833#comment-16942833
 ] 

Eric Badger commented on YARN-9860:
---

{quote}
You are spot on that this will be an issue. The challenge is that if we mount 
the read-write log dirs into the container and the container user isn't the 
user YARN expects, the writes could fail or YARN may be unable to clean up the 
logs. I talked with Craig on this a bit and he had some interesting thoughts on 
how we might handle it with fuse. For the sake of this patch, I didn't want to 
get bogged down in the details there, given this has enough going already. 
Could we address logging in a follow up? In the meantime, with debug delay 
enabled, doing a docker logs on the exited container will allow admins to take 
a look, since the output redirection typically done by YARN is dropped in 
Service Mode.
{quote}
I'm ok having a followup for this. My comment was mostly for my own 
information. I'm not opposed to the feature, I just think it might be a little 
bit difficult to deal with if there are configuration issues or other issues 
that cause the containers to fail. I think that debug delay to not remove 
containers for awhile after completion could be a useful debugging tool, 
though. 

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-02 Thread Shane Kumpf (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942699#comment-16942699
 ] 

Shane Kumpf commented on YARN-9860:
---

{quote}Can you give more clear definition of service mode?
{quote}
To simplify it the best I can, in Service Mode, YARN does not set the user 
(--user and --group-add) when running the container. The rest of the changes 
are in support of dropping the user in this mode. A simple use case where this 
is needed is running the official postgres image without modification. Note 
that this mode is disabled by default to limit any security implications.
{quote}I don't understand the reason to add --user= parameter only when service 
mode is enabled.
{quote}
That code does the opposite of what you stated, user is only passed when 
service mode is NOT enabled, which is what we want.
{quote}If there are no log directories, how would you attack debugging 
container failures?
{quote}
You are spot on that this will be an issue. The challenge is that if we mount 
the read-write log dirs into the container and the container user isn't the 
user YARN expects, the writes could fail or YARN may be unable to clean up the 
logs. I talked with Craig on this a bit and he had some interesting thoughts on 
how we might handle it with fuse. For the sake of this patch, I didn't want to 
get bogged down in the details there, given this has enough going already. 
Could we address logging in a follow up? In the meantime, with debug delay 
enabled, doing a {{docker logs}} on the exited container will allow admins to 
take a look, since the output redirection typically done by YARN is dropped in 
Service Mode.

I've done extensive testing of an earlier version of this patch and it 
addresses the use case and works as expected. I'm going to do some additional 
testing today with the patch here to make sure there are no regressions. 

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-01 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942423#comment-16942423
 ] 

Eric Yang commented on YARN-9860:
-

Patch 002 uses System.getenv("USER") to determine the current running user.  
This may grab the 
yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user which is 
different than the user that store files for public resource directory.

{code}
@@ -1640,12 +1675,14 @@ int get_docker_run_command(const char *command_file, 
const struct configuration
   privileged = get_configuration_value("privileged", 
DOCKER_COMMAND_FILE_SECTION, &command_config);
 
   if (privileged == NULL || strcmp(privileged, "false") == 0) {
-char *user_buffer = make_string("--user=%s", user);
-ret = add_to_args(args, user_buffer);
-free(user_buffer);
-if (ret != 0) {
-  ret = BUFFER_TOO_SMALL;
-  goto free_and_exit;
+if (service_mode_enabled != 0) {
+  char *user_buffer = make_string("--user=%s", user);
+  ret = add_to_args(args, user_buffer);
+  free(user_buffer);
+  if (ret != 0) {
+ret = BUFFER_TOO_SMALL;
+goto free_and_exit;
+  }
 }
 no_new_privileges_enabled =
 get_configuration_value("docker.no-new-privileges.enabled",
@@ -1725,9 +1762,11 @@ int get_docker_run_command(const char *command_file, 
const struct configuration
 goto free_and_exit;
   }
{code}

I don't understand the reason to add --user= parameter only when service mode 
is enabled.  Need more clarity on what service mode is, and what it tries to do 
with enabling application visibility.  Is service mode trying to get a root 
like container without giving actual root access?

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-01 Thread Eric Yang (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942403#comment-16942403
 ] 

Eric Yang commented on YARN-9860:
-

[~Prabhu Joseph] [~shaneku...@gmail.com] Can you give more clear definition of 
service mode?

>From the code, I can not tell how configuration are injected, and how is this 
>mode an improvement to what is available.  Thanks

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-10-01 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16942244#comment-16942244
 ] 

Prabhu Joseph commented on YARN-9860:
-

Thanks [~ebadger] for reviewing. Will update you on above comment.




> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-09-28 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939893#comment-16939893
 ] 

Hadoop QA commented on YARN-9860:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 35s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  7m 
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 14s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 36 new + 92 unchanged - 1 fixed = 128 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  3s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 26s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
52s{color} | {color:green} hadoop-yarn-services-core in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
43s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 34s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12981574/YARN-9860-002.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux a31e0ccc6b1a 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-09-27 Thread Eric Badger (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939677#comment-16939677
 ] 

Eric Badger commented on YARN-9860:
---

If there are no log directories, how would you attack debugging container 
failures? 

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-09-27 Thread Prabhu Joseph (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939534#comment-16939534
 ] 

Prabhu Joseph commented on YARN-9860:
-

[~skumpf] [~sunilg] Can you review this Jira when you get time. This provides 
service mode for Docker containers on YARN. The testcase failures are not 
related and have raised YARN-9862 to fix the same. Thanks.

> Enable service mode for Docker containers on YARN
> -
>
> Key: YARN-9860
> URL: https://issues.apache.org/jira/browse/YARN-9860
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9860-001.patch, YARN-9860-002.patch
>
>
> This task is to add support to YARN for running Docker containers in "Service 
> Mode". 
> Service Mode - Run the container as defined by the image, but still allow for 
> injecting configuration. 
> Background:
>   Entrypoint mode helped - now able to use the ENV and ENTRYPOINT/CMD as 
> defined in the image. However, still requires modification to official images 
> due to user propagation
> User propagation is problematic for running a secure cluster with sssd
>   
> Implementation:
>   Must be enabled via c-e.cfg (example: docker.service-mode.allowed=true)
>   Must be requested at runtime - (example: 
> YARN_CONTAINER_RUNTIME_DOCKER_SERVICE_MODE=true)
>   Entrypoint mode is default enabled for this mode (If Service Mode is 
> requested, YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE should be set 
> to true)
>   Writable log mount will not be added - stdout logging may still work 
> with entrypoint mode - remove the writable bind mounts
>   User and groups will not be propagated (now: docker run --user nobody 
> --group-add=nobody  , after: docker run  )
>   Read-only resources mounted at the file level, files get chmod 777, 
> parent directory only accessible by the run as user.
> cc [~shaneku...@gmail.com]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

2019-09-27 Thread Hadoop QA (Jira)



[ 
https://issues.apache.org/jira/browse/YARN-9860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16939354#comment-16939354
 ] 

Hadoop QA commented on YARN-9860:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 50s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m  
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
14s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m 17s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 5 new + 92 unchanged - 1 fixed = 97 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m  5s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m  
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 
28s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 20s{color} 
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
41s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 57s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:efed4450bf1 |
| JIRA Issue | YARN-9860 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12981534/YARN-9860-001.patch |
| Optional Tests |  dupname  asflicense  compile  javac  javadoc  mvninstall  
mvnsite  unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux 27658fe680d3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 8a9ede5 |
| maven | versi

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

[jira] [Commented] (YARN-9860) Enable service mode for Docker containers on YARN

31 matches

Site Navigation

Mail list logo

Footer information