[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16547568#comment-16547568
]
Aljoscha Krettek commented on YARN-7590:
Thanks a lot [~ebadger]! This was indeed the problem. I thought it might have
been a problem with the setuid/permissions setup that's why I didn't check.
FYI, this is not a production cluster but a little testing project for setting
up a distributed kerberized cluster on Docker:
https://github.com/aljoscha/docker-hadoop-secure-cluster.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 2.6.6, 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch, YARN-7590.branch-2.000.patch,
> YARN-7590.branch-2.6.000.patch, YARN-7590.branch-2.7.000.patch,
> YARN-7590.branch-2.8.000.patch, YARN-7590.branch-2.9.000.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546828#comment-16546828
]
Eric Badger commented on YARN-7590:
---
Is your NM running as root?
{noformat}
if (caller_uid != info.st_uid) {
fprintf(LOGFILE, "Permission mismatch for %s for caller uid: %d, owner uid:
%d.\n", nm_root, caller_uid, info.st_uid);
return 1;
}
{noformat}
Looks like you're running into this error, and caller_uid is set to 0.
caller_uid is the first argument to check_nm_local_dir, which is always called
with nm_uid as its first argument. So to me that looks like the NM is being run
as root
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 2.6.6, 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch, YARN-7590.branch-2.000.patch,
> YARN-7590.branch-2.6.000.patch, YARN-7590.branch-2.7.000.patch,
> YARN-7590.branch-2.8.000.patch, YARN-7590.branch-2.9.000.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16546289#comment-16546289
]
Aljoscha Krettek commented on YARN-7590:
Hi,
I just came across this issue. I have a kerberized YARN cluster setup that used
to work with Hadoop 2.8.3. Now I'm getting the following error:
{code}
main : run as user is hadoop-user
main : requested yarn user is hadoop-user
Permission mismatch for /hadoop-data/nm-local-dirs for caller uid: 0, owner
uid: 1001.
Couldn't get userdir directory for hadoop-user.
{code}
{{hadoop-user}} is the user that I want to use to run my application, {{0}} is
the uid of {{root}}, {{1001}} is the uid of the {{yarn}} user. {{hadoop-user}}
is only in the group {{hadoop-user}}, {{yarn}} is in the groups ({{yarn}},
{{hadoop}}).
{{container-executor}} has these permissions:
{code}
---Sr-s--- 1 root yarn 234175 May 8 02:58 container-executor
{code}
{{container-executor.cfg}} has these permissions:
{code}
-r 1 root yarn 208 Jul 17 08:20 container-executor.cfg
{code}
My directories have these permissions:
{code}
root@slave1:/hadoop-data# ls -lah
total 16K
drwxr-xr-x 1 yarn yarn 4.0K Jul 17 08:33 .
drwxr-xr-x 1 root root 4.0K Jul 17 08:37 ..
drwxr-xr-x 1 yarn yarn 4.0K Jul 17 08:33 nm-local-dirs
drwxr-xr-x 1 yarn yarn 4.0K Jul 17 08:33 nm-log-dirs
{code}
Anyone know what could be causing this? Any help is greatly appreciated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 2.6.6, 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch, YARN-7590.branch-2.000.patch,
> YARN-7590.branch-2.6.000.patch, YARN-7590.branch-2.7.000.patch,
> YARN-7590.branch-2.8.000.patch, YARN-7590.branch-2.9.000.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331454#comment-16331454
]
Eric Yang commented on YARN-7590:
-
Thank you [~miklos.szeg...@cloudera.com]
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 2.6.6, 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch, YARN-7590.branch-2.000.patch,
> YARN-7590.branch-2.6.000.patch, YARN-7590.branch-2.7.000.patch,
> YARN-7590.branch-2.8.000.patch, YARN-7590.branch-2.9.000.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331284#comment-16331284
]
Miklos Szegedi commented on YARN-7590:
--
Thank you, [~eyang] for the patches, I verified and committed them.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Fix For: 2.6.6, 3.1.0, 2.10.0, 2.9.1, 3.0.1, 2.8.4, 2.7.6
>
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch, YARN-7590.branch-2.000.patch,
> YARN-7590.branch-2.6.000.patch, YARN-7590.branch-2.7.000.patch,
> YARN-7590.branch-2.8.000.patch, YARN-7590.branch-2.9.000.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16325345#comment-16325345
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Sure, I'll add 2.x patches on Tuesday.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Fix For: 3.1.0, 2.10.0, 3.0.1
>
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322776#comment-16322776
]
Miklos Szegedi commented on YARN-7590:
--
Thank you for the contribution [~eyang]! I am still working on the branch-2
backport. It is already committed to trunk & branch-3.0.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322755#comment-16322755
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Thank you for the review and commit. :)
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322726#comment-16322726
]
Hudson commented on YARN-7590:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13482 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13482/])
YARN-7590. Improve container-executor validation check. Contributed by
(szegedim: rev bc285da107bb84a3c60c5224369d7398a41db2d8)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322705#comment-16322705
]
Miklos Szegedi commented on YARN-7590:
--
+1. I will commit this shortly.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322681#comment-16322681
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 48s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
22s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905712/YARN-7590.010.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 595f4bd2b69f 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2e0a451 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19199/testReport/ |
| Max. process+thread count | 339 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19199/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch, YARN-7590.010.patch
>
>
> There is minimum check for prefix path for container-executor.
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16322538#comment-16322538
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Sure, I can wait for YARN-7705 and do the
update. Thanks for the heads up.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321664#comment-16321664
]
Miklos Szegedi commented on YARN-7590:
--
[~eyang], The code I suggested above
{code}
fprintf(LOGFILE, "Error checking file stats for %s %d %s.\n", nm_root, err,
strerror(err));
{code}
It should be the following:
{code}
fprintf(LOGFILE, "Error checking file stats for %s %d %s.\n", nm_root, err,
strerror(errno));
{code}
This is my mistake, I apologize. Please update the patch. Also I am inclined to
wait until YARN-7705 gets checked in and update this patch to call your new
function there also. What do you think?
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321590#comment-16321590
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
29m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12905587/YARN-7590.009.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 786ebadcdefc 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 12d0645 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19194/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19194/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch,
> YARN-7590.009.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compro
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321274#comment-16321274
]
Miklos Szegedi commented on YARN-7590:
--
[~eyang], I figured it out.
{code}
char *local_path = "target";
{code}
This path is incomplete. We should use {{TEST_ROOT "target"}} to follow the
standard (see the function above this line) and let's do an mkdirs() to make
sure it exists and the test can be run from any directory. That caused the
failure on my test machine.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16319001#comment-16319001
]
Eric Yang commented on YARN-7590:
-
Hi [~miklos.szeg...@cloudera.com], could you shed some lights on the error that
you encountered with target directory? I am unable to reproduce it.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314329#comment-16314329
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 42s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
36s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904891/YARN-7590.008.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux b6c8c8a0c917 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a81144d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19131/testReport/ |
| Max. process+thread count | 302 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19131/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch, YARN-7590.008.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313730#comment-16313730
]
Miklos Szegedi commented on YARN-7590:
--
[~eyang], thank you for the updated patch.
{code}
Testing check_nm_local_dir()
Error checking file stats for target -1 Unknown error -1.
test_nm_local_dir expected 0 got 1
{code}
I ran the unit test with the latest change and I got the error above.
I also found that you probably do not want to return out of memory here but
another error code:
{code}
int check = check_nm_local_dir(nm_uid, *local_dir_ptr);
if (check != 0) {
container_dir = NULL;
}
if (container_dir == NULL) {
return OUT_OF_MEMORY;
}
{code}
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313623#comment-16313623
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
41s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904830/YARN-7590.007.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux ec293da7637c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 83b513a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19126/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19126/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor t
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16313502#comment-16313502
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Sorry about missing the last point earlier. I
have refined the patch according to your comments. Thank you for the review.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch, YARN-7590.007.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16310534#comment-16310534
]
Miklos Szegedi commented on YARN-7590:
--
Thank you for the patch [~eyang]. I have two more style issues. I also verified
the patch and it runs a basic mapreduce job and does not allow the scenario in
the description as expected.
{code}
fprintf(LOGFILE, "Error checking file stats for %s.\n", nm_root);
{code}
It would be very useful to have a meaningful error message like
{{fprintf(LOGFILE, "Error checking file stats for %s %d %s.\n", nm_root, err,
strerror(err));}}. It helps a lot to support the feature.
{code}
if (check != 0 || strstr(container_log_dir, "..") != 0) {
{code}
Like I mentioned before, I would separate the two checks with a meaningful
error message in the second case. The first one already prints inside the call.
This one also helps to support the feature.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16309902#comment-16309902
]
Eric Yang commented on YARN-7590:
-
Happy New Year [~miklos.szeg...@cloudera.com], Can you review the 006 patch?
Thank you
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16308445#comment-16308445
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
21s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12904249/YARN-7590.006.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 14846bdbd8af 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7fe6f83 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19070/testReport/ |
| Max. process+thread count | 317 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19070/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch,
> YARN-7590.006.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302137#comment-16302137
]
Miklos Szegedi commented on YARN-7590:
--
Thank you for the patch [~eyang]. I have a few style issues:
configuration.c has a new line with the patch that is not needed.
{code}
fprintf(LOGFILE, "Error checking file stats for %s.\n", nm_root);
{code}
It will be helpful to print out the actual error code for debugging.
{code}
fprintf(LOGFILE, "Permission mismatch for %s for uid: %d.\n", nm_root,
caller_uid);
{code}
How about printing {{info.st_uid}} as well?
{code}
if (check != 0 || strstr(container_log_dir, "/../") != 0) {
{code}
It is safer to check for ".." and also this check should be in a separate if
with a proper log message to help debugging.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302020#comment-16302020
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
30m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
58s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 31s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903462/YARN-7590.005.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux a5d83844ddc3 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 52babbb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19019/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19019/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16301970#comment-16301970
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Thank you for the feedback, and I revised the
patch according to your feedback. I am going to take time off for next week.
If there is any improve to be done, let's sync up after the New Year. Merry
Christmas, and Happy New Year.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch, YARN-7590.005.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300387#comment-16300387
]
Miklos Szegedi commented on YARN-7590:
--
Thank you for the patch, [~eyang]. I have a few minor comments left.
{code}
717 int caller_uid = 0;
{code}
Just in case I would have an invalid init value like -1.
{code}
712 int result = check_nm_local_dir(caller_uid, container_log_dir);
713 if (result != 0) {
714 container_log_dir = NULL;
715 }
...
1056int result = check_nm_local_dir(caller_uid, *log_root);
1057if (result != 0) {
1058 app_log_dir = NULL;
1059}
{code}
I am missing here a useful comment like below. You may also want to mention the
faulting directory.
{code}
fprintf(LOGFILE, "Permission mismatch for %s for uid: %d.\n", nm_root,
caller_uid);
{code}
Even better a log in check_nm_local_dir in case of failure would help a lot to
diagnose problems.
{code}
531 int check_nm_local_dir(int caller_uid, const char *nm_root) {
532 struct stat info;
533 stat(nm_root, &info);
534 if (caller_uid != info.st_uid) {
535 return 1;
536 }
537 return 0;
538 }
{code}
There is no error check on the stat call.
{code}
711 char *container_log_dir = get_app_log_directory(*log_dir_ptr,
combined_name);
712 int result = check_nm_local_dir(caller_uid, container_log_dir);
713 if (result != 0) {
714 container_log_dir = NULL;
715 }
{code}
{{create_container_directories()}} needs to check for {{log_dir_ptr}} not
{{container_log_dir}} that does not exist, yet.
Also a note. If the check succeeds, we do an mkdirs() that walks up the stack
and may create parent directories. It may be good to put the check into mkdirs
as well (or only there), when we need to create a directory.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is owned by the same user as the caller to
> container-executor.
> # Make sure the log directory prefix is owned by the same user as the caller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299455#comment-16299455
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
30s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903145/YARN-7590.004.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 3f04492624ad 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 5ab632b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19002/testReport/ |
| Max. process+thread count | 330 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19002/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch, YARN-7590.004.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/b
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16299272#comment-16299272
]
genericqa commented on YARN-7590:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 12s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903122/YARN-7590.003.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 079354060f13 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 382215c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/19001/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19001/testReport/ |
| Max. process+thread count | 303 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19001/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch,
> YARN-7590.003.patch
>
>
> There is minimum check for prefix path for cont
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298990#comment-16298990
]
Miklos Szegedi commented on YARN-7590:
--
Thank you for the patch, [~eyang].
I see two more issues.
{{uid}} could just be a global variable saving some code but using locals is
fine. However, we have now a caller uid, a yarn uid and a run as uid. Please
rename the uid you created as you pass along the functions as caller_uid.
Also, the patch does not apply to the scenario in the initial description.
Please do the check in {{create_log_dirs}} as well.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16291770#comment-16291770
]
Eric Yang commented on YARN-7590:
-
The unit test failure for TestContainerLaunch is caused by YARN-7381, not
related to this patch.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290182#comment-16290182
]
genericqa commented on YARN-7590:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 1s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 34s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 36s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12901965/YARN-7590.002.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 56949a81f277 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 46e18c8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18913/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18913/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18913/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
>
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290142#comment-16290142
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Thank you for the review. I revised the code to
preserve uid during program startup per your suggestion. Let me know if this
works. Thanks.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch, YARN-7590.002.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288599#comment-16288599
]
Miklos Szegedi commented on YARN-7590:
--
[~eyang], the first line of {{main()}} calls {{assert_valid_setup()}} that
calls {{setuid(0)}}. You need to sample the yarn uid with {{getuid()}} and
store before this call to avoid the following error:
{code}
515 uid 2002 gid 2002 euid 0 egid 2002
517 uid 0 gid 2002 euid 0 egid 2002
main : command provided 0
main : run as user is nobody
main : requested yarn user is foo
521 uid 0 gid 2002 euid 0 egid 2002
556 uid 0 gid 2002 euid 0 egid 2002
uid 0 gid 2002 euid 0 egid 2002
558 uid 0 gid 2002 euid 99 egid 99
Permission mismatch for /tmp/hadoop-foo/nm-local-dir for uid: 0.
{code}
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284372#comment-16284372
]
Miklos Szegedi commented on YARN-7590:
--
[~eyang], sorry about the delay. Due to the sensitivity of the issue I intend
to do some end to end tests but I did not get there yet.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16283906#comment-16283906
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] Hi Miklos, would you mind to review this patch?
Thanks
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277725#comment-16277725
]
genericqa commented on YARN-7590:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
27m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 8s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 17m
19s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7590 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900561/YARN-7590.001.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 4e115e04cfd1 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d8863fc |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18784/testReport/ |
| Max. process+thread count | 342 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18784/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7590.001.patch
>
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277252#comment-16277252
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] getuid() may produce uid belong to multiple
parties because the given permission is yarn group. If the check make sure
that uid and node manager prefix directory uid are consistent, then the
validation might be sufficient. At minimum, other yarn group users can not
puncture holes on the file system. Thanks for the suggestion.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275315#comment-16275315
]
Miklos Szegedi commented on YARN-7590:
--
[~eyang], why do not we just call getuid() to get the uid?
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275300#comment-16275300
]
Eric Yang commented on YARN-7590:
-
[~miklos.szeg...@cloudera.com] One problem with option 3 is that container
executor only knows about yarn gid, not uid. Is it sufficient to check against
{{yarn.nodemanager.linux-container-executor.group}} to determine if the prefix
directory is trusted? The answer is probably not because Hadoop configuration
directory could be set to hadoop group, which is the same group as
{{yarn.nodemanager.linux-container-executor.group}}.
Conversely, we can introduce another parameter in container-executor.cfg to
identify yarn uid for validating the prefix directory must owned by yarn uid.
Are we ok with this change?
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Affects Versions: 2.0.1-alpha, 2.2.0, 2.3.0, 2.4.0, 2.5.0, 2.6.0, 2.7.0,
> 2.8.0, 2.8.1, 3.0.0-beta1
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275232#comment-16275232
]
Andrew Wang commented on YARN-7590:
---
Thanks Eric!
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275221#comment-16275221
]
Eric Yang commented on YARN-7590:
-
[~ebadger]
{quote}
I've brought this up in the past (can't remember where) and it didn't get
anywhere. I believe there was a reason that we didn't want yarn-site.xml to be
owned by root. Possibly because it would break current deploys?
{quote}
I don't think there is a hard requirement that yarn-site.xml must be owned by
yarn user. This may have been miscommunication. My clusters have been using
root:hadoop, 644 for yarn-site.xml for most of the past 5 years.
[~miklos.szeg...@cloudera.com] +1 on option 3. It is smart and safe way to
validate the prefix directory with minimum amount of code change.
[~andrew.wang] This JIRA assumes YARN is compromised. Theoretical
interpretation doesn't make this reality yet. I don't believe this is a
blocker. Versions are set accordingly.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275205#comment-16275205
]
Andrew Wang commented on YARN-7590:
---
Hi folks, is this a release blocker? Could someone set the affects versions for
tracking?
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275168#comment-16275168
]
Mike Yoder commented on YARN-7590:
--
{quote}
Container executor should link to a C based XML parser to get local directories
from yarn-site.xml.
{quote}
Setuid/setgid binaries are super-dangerous, and the target of attackers. The
less logic we can put in them, the better. Putting an xml parser in there...
eeek.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275142#comment-16275142
]
Miklos Szegedi commented on YARN-7590:
--
I have two more options:
3. Instead of getting a prefix path from container-executor.cfg and/or
yarn-site.xml you could check, if yarn has permissions to the desired path and
all its parents. There is no need to check either of the config files in this
case, so this would be the simplest change.
4. Disallow disruptive changes: check, if container-executor is about to chmod
an existing directory with incompatible permissions and disallow it.
I am in favor of 2. or 3.
There are multiple reasons why currently it is not a good idea to call out to
yarn-site.xml from container-executor (Option 1.):
1. XML parsing may add yet another library that increases the attack surface
2. You need to make sure (--checksetup) that the XML has the right permissions
3. CLASSPATH is not inherited, so it may pick up a different yarn-site.xml than
what the node manager uses
4. Potentially breaking change: requiring yarn-site.xml parents writable only
by root
5. Potentially breaking change: non-root users can no longer modify
yarn-site.xml settings
I am all in favor of simple configuration provided by option 1., but at this
time I would suggest having a separate config line in container-executor.cfg
(option 2.) or option 3.. A future compatibility breaking JIRA can merge the
two config files properly implementing proper rights checks. container-executor
could give a proper error message in case of option 2., so that the admin can
update the directories in case of a failure.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275068#comment-16275068
]
Eric Badger commented on YARN-7590:
---
bq. Make sure the prefix path is same as the one in yarn-site.xml, and
yarn-site.xml is owned by root, 644, and marked as final in property.
I've brought this up in the past (can't remember where) and it didn't get
anywhere. I believe there was a reason that we didn't want yarn-site.xml to be
owned by root. Possibly because it would break current deploys?
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7590) Improve container-executor validation check
[
https://issues.apache.org/jira/browse/YARN-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274708#comment-16274708
]
Eric Yang commented on YARN-7590:
-
There is currently two proposals to address this issue:
h3. Proposals
# Container executor should link to a C based XML parser to get local
directories from yarn-site.xml.
# Add configuration to container executor config for local directories for
container executor to verify allowed prefix path.
h3. Obstacle
If we choose option 1, expat and libxml2 are license compatible libraries for
this purpose. However, both parsers had security vulnerability as well that
allow hijack of doctype to connect to remote server for DTD validation. The
implementation must disable remote schema validation.
If we choose option 2, this design was originally proposed 6+ years ago, but
implementation was lost in MAPREDUCE-2413. If we put the duplicated properties
on separate files, then it is likely to get lost during code optimization
again. I recommend to avoid this path.
> Improve container-executor validation check
> ---
>
> Key: YARN-7590
> URL: https://issues.apache.org/jira/browse/YARN-7590
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: security, yarn
>Reporter: Eric Yang
>
> There is minimum check for prefix path for container-executor. If YARN is
> compromised, attacker can use container-executor to change system files
> ownership:
> {code}
> /usr/local/hadoop/bin/container-executor spark yarn 0 etc /home/yarn/tokens
> /home/spark / ls
> {code}
> This will change /etc to be owned by spark user:
> {code}
> # ls -ld /etc
> drwxr-s---. 110 spark hadoop 8192 Nov 21 20:00 /etc
> {code}
> Spark user can rewrite /etc files to gain more access. We can improve this
> with additional check in container-executor:
> # Make sure the prefix path is same as the one in yarn-site.xml, and
> yarn-site.xml is owned by root, 644, and marked as final in property.
> # Make sure the user path is not a symlink, usercache is not a symlink.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
