[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242631#comment-16242631
]
Eric Yang commented on YARN-7197:
-
[~shaneku...@gmail.com]
{quote}
I'm aware of the differences between mounting the socket and true "docker in
docker" as well as the impact of privileged containers. My point was that not
allowing arbitrary destination paths within the container for mounts makes some
use cases more difficult than needed. HBase running in a container that expects
to find /etc/hbase/conf/hbase-site.xml and we want to mount that file from the
distributed cache, if you'd prefer.
{quote}
System configuration is safer than user's own configuration because system
administrator is the only one who optimized the properties for the target
hardware. User's configuration as superset of configuration is a nice to have
feature for developer that knows their data well. I don't object to have both.
I am not in favor for creating many bind-mount paths for black listed items
when user did not ask for them. There is too many down side to create
obstruction inside container in the name of security.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242605#comment-16242605
]
Shane Kumpf commented on YARN-7197:
---
I'm aware of the differences between mounting the socket and true "docker in
docker" as well as the impact of privileged containers. My point was that not
allowing arbitrary destination paths within the container for mounts makes some
use cases more difficult than needed. HBase running in a container that expects
to find {{/etc/hbase/conf/hbase-site.xml}} and we want to mount that file from
the distributed cache, if you'd prefer.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242583#comment-16242583
]
Eric Yang commented on YARN-7197:
-
[~shaneku...@gmail.com]
{quote}
Unfortunately, I've recently encountered a scenario where this restriction
would have made the use case pretty difficult to get working. The issue goes
back to what I mentioned about systemd and API filesystems. There are valid
cases for mounting the docker socket, such as CI. Due to systemd running in the
container, /run is mounted as a tmpfs after the docker bind mount is handled,
hiding /run/docker.sock in the container, so docker in docker use cases that
also use systemd as the init process would not be possible (without
modifications to the docker daemon config on the host). If we do impose that
restriction by default, then we'll also need a way to disable it.
{quote}
There are a lot of painful lesson to run docker in docker. There is a good
blog about
[this|https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/].
I had almost exact experience with the drawbacks.
We can run docker in parallel, which allows privileged container to mount
/run/docker.sock to spawn docker at top level docker image. If a normal user
mount /run/docker.sock into a non-privileged container, they should not have
access to control /run/docker.sock. Today, they may have access to
/run/docker.sock due to a bug for not pass in primary group, and allow
non-privileged user to become root group (YARN-7430). However, this
implementation will not safe guard toward docker instances spawned from inside
the container. It is possible to lose tracking of container spawned by
containers. Hence, be very careful about who you hand the keys over to spawn
privileged containers.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242539#comment-16242539
]
Eric Yang commented on YARN-7197:
-
[~jlowe] {quote}
If the whitelist only allows bindmounts to the same path as the host then I
think we're OK here, but if it doesn't then we need to address that.
{quote}
Most of the docker image is not a full Linux image. It is possible that system
admin provide ability to mount /etc/hadoop/conf to a path of image chosen
directory to read hadoop configuration. It would be better for us to support
mapping of different source to destination path.
If someone mounted {{/etc/shadow}}, {{/etc/passwd}}, {{/etc/group}} and
{{/etc/sudoers}} from their user home directory into container for privileges
escalation. They still need to defeat the following:
# Add sudo binary into the container image.
# Find a way to remove {{--cap-drop=ALL}} which we hard coded into
container-executor.
# Gain write access to outside world through a mounted location like HDFS.
Trusted registry protects 1. Container-executor binary protects 2. Allowed
white list protects 3. I think it's difficult to get a privilege escalation,
if the protections are in place. The original intend to protect QA users from
destroying cluster hosts and giving them access to spawn root container is a
noble cause. However, I don't think we will find a right way to protect root
from root using black list. {{--cap-drop=ALL}} is the better way to give them
container root power and keep that access within the container. I will leave
this JIRA open for others to try.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242481#comment-16242481
]
Shane Kumpf commented on YARN-7197:
---
{quote}
Would it be too restrictive to enforce that a whitelist mount must have the
same path in the image as it does in the host? Then we can't have users
swapping out /etc for some custom directory. I looked briefly at the current
whitelist mount support and that appears to be how it works.
{quote}
Unfortunately, I've recently encountered a scenario where this restriction
would have made the use case pretty difficult to get working. The issue goes
back to what I mentioned about systemd and API filesystems. There are valid
cases for mounting the docker socket, such as CI. Due to systemd running in the
container, {{/run}} is mounted as a tmpfs after the docker bind mount is
handled, _hiding_ {{/run/docker.sock}} in the container, so _docker in docker_
use cases that also use systemd as the init process would not be possible
(without modifications to the docker daemon config on the host). If we do
impose that restriction by default, then we'll also need a way to disable it.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242415#comment-16242415 ] Jason Lowe commented on YARN-7197: -- bq. Docker image can also be regulated through trusted docker registry. This makes degree of difficulty to crack the system harder. Agreed, if we can trust the images themselves aren't a source of nefarious setup then we need to make sure the whitelist isn't abused to turn a trusted image into an untrusted one. Would it be too restrictive to enforce that a whitelist mount must have the same path in the image as it does in the host? Then we can't have users swapping out /etc for some custom directory. I looked briefly at the current whitelist mount support and that appears to be how it works. bq. I am starting to doubt the feasibility of blacklist approach to prevent jailbreak or leak information at the cost of creating many bind mounted location. Totally agree, security is hard to get right. I don't see this as the only way to prevent jailbreaks or leaks, rather just another tool in the admin's toolbox to help make the whitelist easier to maintain if they have a case where they want/need to whitelist something above another path that is sensitive. I would expect many admins to never use it and instead be extremely selective in what they whitelist, including many not whitelisting anything. bq. We should accept the fact that privileged container is equal to root power on the host system, hence only trusted users can be given access to spawn privileged containers. We only govern the system by checking sudo privileges to spawn privileged container, and honor file system ACL. If we prevent the user from providing arbitrary images then I think we're close to this, but as you pointed out if the user can bash an critical path in the image with a bind mount then an unprivileged user could escalate within the image (e.g.: smash suodoers file and image has sudo binary, smash /etc and image has su binary, etc.). If the whitelist only allows bindmounts to the same path as the host then I think we're OK here, but if it doesn't then we need to address that. bq. All bind mount path should not exceed the run command buffer. We have to address that problem even with the current whitelist support. Filed YARN-7455 to track that issue. So to sum up, I agree this JIRA becomes simply a "nice to have" and not necessary if whitelisted mounts cannot be retargeted within the image. I'm not sure we have to close it as won't fix, since this could be a useful tool for admins that know what they're doing. However I wouldn't see it as a must fix for the next release either. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch, > YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241479#comment-16241479 ] Eric Yang commented on YARN-7197: - I am starting to doubt the feasibility of blacklist approach to prevent jailbreak or leak information at the cost of creating many bind mounted location. The permutation is too costly, and the solution is difficult to use for developer and system admin cross examine every files and symlinks/hard links in the system. We should accept the fact that privileged container is equal to root power on the host system, hence only trusted users can be given access to spawn privileged containers. We only govern the system by checking sudo privileges to spawn privileged container, and honor file system ACL. Although the rules may sound like a good idea, but computation of directory depth * number of black listed item * user defined mounts * symlink permutation * hardlink permutation can be a challenge. All bind mount path should not exceed the run command buffer. I think this solution does not scale in real life. Hence, I propose that we shift focus toward YARN-7221, YARN-7446 and YARN-7430. If group is enforced properly, and sudoers check are verified, then there is much less risk to expose host information into container container or let unauthorized user to take control of the host system. If the community agree with this analysis, I will close this as won't fix. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch, > YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241257#comment-16241257
]
Eric Yang commented on YARN-7197:
-
{quote}
{code}
docker run -it -v /etc:/home/test/etc --mount
'type=bind,source=/var/empty/sshd,target=/home/test/etc/shadow,readonly'
centos:latest bash
{code}
{quote}
This case will fail, unless developer change their rules to be more specific of
which sub-directory that they like to mount. If I make empty file and sockets,
and mount those, it seems a bit over kill, and harder to secure because yarn
user isn't root. There might be some limitation to make the same ownership
file in yarn working directory to map to container. Therefore, we fail fast
and let developer and system admin resolve this on their own. Is this a fair
compromise?
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241233#comment-16241233
]
Eric Yang commented on YARN-7197:
-
[~jlowe]
{quote}
If I'm understanding properly, we need to be worried not only about what host
path the user is trying to mount but also where they are trying to mount it
within the image? I was under the impression the contents of the image were
user-provided. If that's the case then I do not see why the user needs a
bind-mount from the host to clobber some path in the image. They could just
create the nefarious contents in the image directly. Am I misunderstanding, or
do you have an example where it is necessary?
{quote}
There were two goals that we try to solve:
# Prevent leaking host sensitive information into docker container, like
{{/etc/shadow}}, {{/etc/sssd}}.
# Prevent docker container to take control of the host, like
{{/var/run/docker.socket}}, raw devices.
Bind mount is to prevent making holes on host system. If the specified path
doesn't exist, -v flag will make the directory on host system. Bind mount will
fail to start container. Users can do almost everything inside the container,
without violation of the above rules. If they created their own socket or raw
devices in container, their creation have no side effect in the host system.
We will not prevent them from doing things like:
{code}
docker run -it -v /home/test/shadow:/etc/shadow centos:latest bash
{code}
We only prevent access, if they attempt to dance around with blacklisted items.
Given that {{/etc}}, {{/run}} are in white list, and {{/etc/shadow}},
{{/run/docker.sock}} are blacklisted.
h3. Attempt to mount black list item to same path in container:
{code}
docker run -it -v /etc/shadow:/etc/shadow centos:latest bash
{code}
System try to launch container with command that would fail:
{code}
docker run -it --mount
'type=bind,source=/var/empty/sshd,target=/etc/shadow,readonly' centos:latest
bash
{code}
h3. Hacker attempts to take control of host docker.
{code}
docker run -it -v /run/docker.sock:/run/docker.sock centos:latest bash
{code}
System will launch container with added blacklisted path:
{code}
docker run -it --mount
'type=bind,source=/var/empty/sshd,target=/run/docker.sock,readonly'
centos:latest bash
{code}
The proposal to protect remapped path will safe guard the following:
h3. Attempt to override blacklisted path to alternate location:
{code}
docker run -it -v /etc/shadow:/home/test/shadow centos:latest bash
{code}
System will launch container with command that will succeed if
{{/home/test/shadow}} doesn't exist:
{code}
docker run -it --mount
'type=bind,source=/var/empty/sshd:/home/test/shadow,readonly' centos:latest bash
{code}
h3. Attempt to mount {{/etc}} to alternate location for development purpose:
{code}
docker run -it -v /etc:/home/test/etc centos:latest bash
{code}
System will launch container with added blacklisted paths:
{code}
docker run -it -v /etc:/home/test/etc --mount
'type=bind,source=/var/empty/sshd,target=/home/test/etc/shadow,readonly'
centos:latest bash
{code}
Does this design look reasonable?
Docker image can also be regulated through trusted docker registry. This makes
degree of difficulty to crack the system harder.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241014#comment-16241014 ] Jason Lowe commented on YARN-7197: -- bq. I see that I missed a key point about mounting above parent directory. When the target location is set to another location, the blacklist addictive does not enforce blacklisted path relative to target location. If I'm understanding properly, we need to be worried not only about what host path the user is trying to mount but also _where_ they are trying to mount it within the image? I was under the impression the contents of the image were user-provided. If that's the case then I do not see why the user needs a bind-mount from the host to clobber some path in the image. They could just create the nefarious contents in the image directly. Am I misunderstanding, or do you have an example where it is necessary? bq. Do you agree that by tracking blacklisted path relative location to target location, we can satisfy the original motive of preventing jail break out of container? I thought the point of this JIRA was to prevent exposing sensitive paths on the host to the container so it's inaccessible even if the user gains privilege within the container. It sounds like you're proposing that the blacklist should also prevent those paths in the image from being clobbered by mounts from the host. If the user controls the image contents then I'm not sure that protects us from much. Apologies if I'm misunderstanding what is meant by "relative location to target location." > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch, > YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240946#comment-16240946
]
Eric Yang commented on YARN-7197:
-
{quote}
Couldn't the same be said about directories? If the admin blacklists /var/run
but whitelists /var or blacklists /dev/pts but allows /dev there can be
problems. The admin needs to understand the ramifications of what they are
whitelisting and blacklisting with respect to what the container needs to
function properly.
The current patch allows users to mount above blacklisted paths and still
access them which I think we all agree defeats the point of the blacklist and
isn't what an admin would expect when configuring the blacklist. Bottom line is
either we need to try to blot out the blacklisted paths when the user mounts
above it regardless of path type or we don't allow admins to blacklist paths
that aren't directories on the host.
{quote}
I see that I missed a key point about mounting above parent directory. When
the target location is set to another location, the blacklist addictive does
not enforce blacklisted path relative to target location. I will make
refinement for this point.
In general, user might get a read-only black listed directory when mount point
are covering large span of paths. If they encounter the error message, they
are likely to ask system admin about the black listed path. They can refine
mount points accordingly. Hackers will try both explicit defined mount or
implicit defined mounts to dodge security. They should not get through using
either source=target path, or source=~target attempts. If this is implemented
accordingly, empty file or socket type are probably lesser concern because
target location usually don't exist.
Do you agree that by tracking blacklisted path relative location to target
location, we can satisfy the original motive of preventing jail break out of
container?
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240765#comment-16240765 ] Jason Lowe commented on YARN-7197: -- bq. Even if it worked, we could be leaking private container information to outside. For example, if someone mount a random number generator socket for encryption seed, and admin blacklisted random number generator from mounting. The result all end up with 0s. There is no one know about a mistake have been made. It could have serious consequence to security. Couldn't the same be said about directories? If the admin blacklists /var/run but whitelists /var or blacklists /dev/pts but allows /dev there can be problems. The admin needs to understand the ramifications of what they are whitelisting and blacklisting with respect to what the container needs to function properly. The current patch allows users to mount above blacklisted paths and still access them which I think we all agree defeats the point of the blacklist and isn't what an admin would expect when configuring the blacklist. Bottom line is either we need to try to blot out the blacklisted paths when the user mounts above it regardless of path type or we don't allow admins to blacklist paths that aren't directories on the host. I'm OK if we want to go with the latter as long as we document that blacklisted paths that are not directories on the host do not work as expected in practice. We can consider adding the support later to remove that restriction. In any case, I think the patch needs to behave like the following: # If a blacklisted path or a path under the blacklisted path attempts to be mounted the container should just fail with an error message stating the requested path is blacklisted. # If a whitelisted requested mount is a prefix of any of the blacklisted paths then the blacklisted path needs a bind-mount to prevent access to the blacklisted path under the requested mount. bq. bind_mount is type-bind,source=, target=, and readonly and two %s Ah sorry, my apologies. I completely missed that bind_mount wasn't a path as the variable name implied but actually the format string for sprintf. This should be passed to sprintf as the format string rather than duplicating it, otherwise someone is going to come along and update one without updating the other. The overrun potential with tmp_buffer (as opposed to tmp_buffer_2) is still a concern. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch, > YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240684#comment-16240684
]
Eric Yang commented on YARN-7197:
-
[~jlowe]
{quote}
Yes, by "explode" I mean the OS will fail the container creation, and that's
where the problem lies in some cases. Here's an example scenario:
# Admin has configured /etc in the whitelist and /etc/shadow in the blacklist.
# User runs a container that bind-mounts /etc but not for malicious reasons
# YARN needs to bind-mount over /etc/shadow to prevent the visibility of that
file on the host from within the container
# Attempting to bind-mount a directory over a file fails, and the container
does not launch
The current patch doesn't handle this case properly because it misses the need
to bind-mount to a blacklisted path when a parent of a blacklisted path is
mounted. Even if it did catch that case, the blacklisted path on the host needs
to be a directory or bind-mounting the empty directory onto the host file is
going to fail, preventing the container from launching. That's what I meant by
stating we need to support bind-mounting an empty file or we're effectively
only supporting blacklisted paths that are directories.
{quote}
I see that I missed your points earlier. You would like the container to start
as much as possible rather than fail early with denied of entry. However, we
don't know for certain that mounting a empty file would not cause eruption to
the container later. There is some risk involved by allowing empty file to be
in place of black listed file that we may not fully understood the risk
involved. For example, {{/etc/shadow}} is mounted from outside empty file, we
end up strange behavior:
{code}
[root@fa79f56310e6 etc]# useradd eyang
useradd: failure while writing changes to /etc/shadow
{code}
Even if it worked, we could be leaking private container information to
outside. For example, if someone mount a random number generator socket for
encryption seed, and admin blacklisted random number generator from mounting.
The result all end up with 0s. There is no one know about a mistake have been
made. It could have serious consequence to security.
The current implementation is more brute force to denied of access, and black
listed directory is read-only, hence it notifies the user early about the
problems rather than failing with silent behavior. Let's say {{/etc}} is white
list, and {{/etc/hadoop}}, {{/etc/passwd}}, {{/etc/group}} are required for
container to have Hadoop configuration and user group information. User can
explicitly specify each item, and admin only required to define {{/etc}} once
in white list. If user attempt to mount {{/etc/shadow}} or {{/etc/}} blindly,
then it will end with denied system call. I think the current proposal is more
safe in terms of usability while preserve required policy in place.
{quote}
The size of tmp_buffer_2 is computed as the lengths of three strings sans their
NUL terminators. Those same three strings are copied into tmp_buffer_2 along
with the constant portion of the format string. sprintf is not allocating a new
destination buffer here. tmp_buffer_2 needs to be big enough to hold not only
the contents of empty_dir, mount_target, and bind_mount but also
type=bind,source= and ,target= and ,readonly along with a terminating NUL. All
of those constant characters in the format string are not part of the size
calculation for tmp_buffer_2 which means the sprintf is going to blow past the
end of the allocated buffer and corrupt the heap. In addition the call to
quote_and_append_arg could blow past the end of tmp_buffer since tmp_buffer is
only a 1K buffer and the paths involved could be quite long.
{quote}
{{bind_mount}} is {{type-bind,source=}}, {{target=}}, and {{readonly}} and two
{{%s}}. The allocated length is 38 char instead of the required 34 in the
alloc_and_clear_memory. With sprintf appending null terminator, it will end up
with 35 characters. The required length for {{empty_dir}} and {{mount_target}}
are already factored in the calculation. Hence, the proper size have already
been accounted. Let me know if I am missing the point. Thanks
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240440#comment-16240440
]
Jason Lowe commented on YARN-7197:
--
bq. Explode might be exaggeration.
Yes, by "explode" I mean the OS will fail the container creation, and that's
where the problem lies in some cases. Here's an example scenario:
# Admin has configured {{/etc}} in the whitelist and {{/etc/shadow}} in the
blacklist.
# User runs a container that bind-mounts {{/etc}} but not for malicious reasons
# YARN needs to bind-mount over {{/etc/shadow}} to prevent the visibility of
that file on the host from within the container
# Attempting to bind-mount a directory over a file fails, and the container
does not launch
The current patch doesn't handle this case properly because it misses the need
to bind-mount to a blacklisted path when a parent of a blacklisted path is
mounted. Even if it did catch that case, the blacklisted path on the host
needs to be a directory or bind-mounting the empty directory onto the host file
is going to fail, preventing the container from launching. That's what I meant
by stating we need to support bind-mounting an empty file or we're effectively
only supporting blacklisted paths that are directories.
bq. File creation is easy, and it is easy to get it wrong too. Someone on the
host OS might add additional files into files in /tmp or get erased by accident.
This isn't an issue if we create the file in the YARN container's working
directory, and that also conveniently solves the cleanup issue when the
container completes.
bq. The default umask might not be the same as OS default, therefore extra
ownership system calls are required to secure the newly created files. Those
operation can slow down the system.
This is adding two system calls to the container executor, a creat followed by
a chmod system call. The slowdown will not be perceptible unless there are I/O
issues with the container's working directory, and if that's the case then
there are going to be many more issues than this as the container tries to run.
bq. I think %s is extra characters counted toward required length for NULL.
The %s characters are not accounted for either. Here's the relevant code again:
{code}
tmp_buffer_2 = (char *) alloc_and_clear_memory(strlen(empty_dir) +
strlen(mount_target) + strlen(bind_mount), sizeof(char));
sprintf(tmp_buffer_2, "type=bind,source=%s,target=%s,readonly",
empty_dir, mount_target);
quote_and_append_arg(_buffer, _buffer_size, "--mount ",
tmp_buffer_2);
{code}
The size of tmp_buffer_2 is computed as the lengths of three strings sans their
NUL terminators. Those same three strings are copied into tmp_buffer_2 along
with the constant portion of the format string. sprintf is not allocating a
new destination buffer here. tmp_buffer_2 needs to be big enough to hold not
only the contents of {{empty_dir}}, {{mount_target}}, and {{bind_mount}} but
also {{type=bind,source=}} and {{,target=}} and {{,readonly}} along with a
terminating NUL. All of those constant characters in the format string are not
part of the size calculation for tmp_buffer_2 which means the sprintf is going
to blow past the end of the allocated buffer and corrupt the heap. In addition
the call to quote_and_append_arg could blow past the end of tmp_buffer since
tmp_buffer is only a 1K buffer and the paths involved could be quite long.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238617#comment-16238617
]
Eric Yang commented on YARN-7197:
-
Hi [~jlowe], thank you for the review.
{quote}
I'm not sure I understand what you mean by "slip through" here.
One of the original blacklist examples you gave above showed /run/docker.socket
in the blacklist, and that's not a directory. That's going to prevent any
mounting above that path because as soon as it tries to mount a directory onto
/run/docker.socket it's going to explode. Similarly the unit test uses
/etc/shadow as an example blacklist path. That prevents mounting /etc into a
container even if /etc is in the whitelist because the container will fail as
soon as Docker tries to mount a directory onto the /etc/shadow file.
{quote}
Explode might be exaggeration. OS prevents the system call from succeeding,
and Docker translate the system exit code with a verbose message. This does
not cause any harm to OS. "Slip through" means mounting jailbreak directory to
target path without OS denied the mount. In the unit test,
{{test_docker_run_banned_mounts}} demonstrated that we can white list /etc, and
black list a child directory (result in mounting a empty directory) and any
file type (they are blocked by OS system call). YARN will clean up container
regardless successful creation or not. Therefore, additional code to check for
something that OS also have check point seems redundant.
{quote}
It is trivial to create these, they do not need to exist outside of the YARN
installation. The nodemanager could create a file in /tmp on startup, chmod it
to a public file, and voila – universal empty file that can be reused across
containers. Or the container-executor could create an empty file within the
container's tmp under its working directory and mount that, which makes it
per-container and gets automatically cleaned up by YARN when the container
exits. Lots of ways to solve this problem.
{quote}
File creation is easy, and it is easy to get it wrong too. Someone on the host
OS might add additional files into files in /tmp or get erased by accident.
The default umask might not be the same as OS default, therefore extra
ownership system calls are required to secure the newly created files. Those
operation can slow down the system. Therefore, I decided to let OS system call
handle mismatch system call instead of adding more logic that might contain
even more holes.
{quote}
It doesn't look like we allocate nearly enough space for tmp_buffer_2 here
since we're not accounting for all of the hardcoded text or terminating NUL:
{quote}
I think {{%s}} is extra characters counted toward required length for NULL.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238528#comment-16238528
]
Jason Lowe commented on YARN-7197:
--
Thanks for updating the patch!
bq. Container-Executor has no prior knowledge of the object type in container
image.
I'm not proposing container-executor know what the object type is in the image,
rather I'm proposing it check the object type on the host. If the type in the
host is a file and in the image it's a dir or vice-versa then it's going to
fail anyway even if it's not blacklisted because we can't mix and match between
dir and non-dir types. (file vs. character device does seem to work, however.)
bq. Directory is the only type that will slip through, and we replace the black
list item with empty directory.
I'm not sure I understand what you mean by "slip through" here.
One of the original blacklist examples you gave above showed /run/docker.socket
in the blacklist, and that's not a directory. That's going to prevent any
mounting above that path because as soon as it tries to mount a directory onto
/run/docker.socket it's going to explode. Similarly the unit test uses
/etc/shadow as an example blacklist path. That prevents mounting /etc into a
container even if /etc is in the whitelist because the container will fail as
soon as Docker tries to mount a directory onto the /etc/shadow file. Going a
directory-only route is preventing any parent directory of a non-directory
blacklisted path from being in the whitelist. Unless that's addressed, we
might as well only support host directories in the blacklist.
bq. We have a universally empty directory for prevent jailbreak, and there is
no universally empty file/socket for prevent jailbreak.
It is trivial to create these, they do not need to exist outside of the YARN
installation. The nodemanager could create a file in /tmp on startup, chmod it
to a public file, and voila -- universal empty file that can be reused across
containers. Or the container-executor could create an empty file within the
container's tmp under its working directory and mount that, which makes it
per-container and gets automatically cleaned up by YARN when the container
exits. Lots of ways to solve this problem.
In the patch, the starts_with method does not need to compute the length of
both strings and figure out which to use. It only needs to compute the length
of the prefix string. strncmp(a, b, strlen(a)) is guaranteed to not be zero if
strlen(b) < strlen(a).
It doesn't look like we allocate nearly enough space for tmp_buffer_2 here
since we're not accounting for all of the hardcoded text or terminating NUL:
{code}
tmp_buffer_2 = (char *) alloc_and_clear_memory(strlen(empty_dir) +
strlen(mount_target) + strlen(bind_mount), sizeof(char));
sprintf(tmp_buffer_2, "type=bind,source=%s,target=%s,readonly",
empty_dir, mount_target);
quote_and_append_arg(_buffer, _buffer_size, "--mount ",
tmp_buffer_2);
{code}
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch, YARN-7197.005.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236421#comment-16236421
]
Hadoop QA commented on YARN-7197:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 16m
34s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895455/YARN-7197.005.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux eb903d16103a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e6ec020 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18312/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18312/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
>Priority: Major
>
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235101#comment-16235101
]
Hadoop QA commented on YARN-7197:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7197 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895308/YARN-7197.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18295/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch, YARN-7197.004.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234551#comment-16234551
]
Hadoop QA commented on YARN-7197:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
34m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 55s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | TEST-cetest |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12895228/YARN-7197.003.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 2f5be0ef2027 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 56b88b0 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18288/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18288/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18288/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL:
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234454#comment-16234454
]
Eric Yang commented on YARN-7197:
-
Hi [~jlowe]
{quote}
If black list contains misconfigured item, i.e, source doesn't exist, container
creation fails.
What's the rationale behind this behavior? I could see admins configuring a
path here that may only exist on some nodes but not all.
{quote}
This is to prevent mistake. If a user misconfigure source path due to typing
mistakes, it could make a empty directory all over the host OS by using -v.
Hence, I took bind mount suggestion to fail fast instead.
{quote}
Seems like we could solve this limitation by either automatically determining
whether the blacklisted path on the host is a file or a directory or having a
way for the admin to give a directive whether the blacklisted path should be
obscured with a file or a directory.
{quote}
Container-Executor has no prior knowledge of the object type in container
image. We depends on docker to determine if there are mismatches and fail
fast. It's more consistent that we let the system to fail fast instead of
letting it slip through. Docker will determine if the mounting source and
target are the same type. Directory is the only type that will slip through,
and we replace the black list item with empty directory. We have a universally
empty directory for prevent jailbreak, and there is no universally empty
file/socket for prevent jailbreak. Therefore, I think it is reasonable to
depend on docker to make this determination, and use empty directory as
fallback on stuff that docker can not catch.
{quote}
Rather than normalizing the banned_mounts every time, would it make more sense
to normalize them just once when we read them in from the config? That would
also remove the need to skip null entries when checking since we could weed
them out during the config read.
check_banned_mounts leaks normalized_path if banned_mounts is null.
check_banned_mounts needs to use strncmp for the length of the banned path to
catch users trying to mount beneath a blacklisted path. For example, /home is
blacklisted by the admin and the user tries to mount /home/yarn.
How do we know mount_src is big enough to do the strcpy of /var/empty/sshd
without overrunning the buffer and scribbling on the heap?
Nit: Checking for a magic value from check_banned_mounts is odd for what is
essentially a boolean predicate function. I'd expect callers to use if (banned)
rather than knowing it's going to return -1 instead of the more natural value,
1.
{quote}
Good catch on the coding style issues. My C skill is a bit rusty, I will fix
those issues.
{quote}
I think we should just fail the container launch if the user tries to mount at
or under a blacklisted path. They clearly are trying to do something the admin
does not want. We should only try to blot out blacklisted paths with an empty
file/dir if and only if the user is trying to mount above a blacklisted path
and could be legitimate behavior. Thoughts?
{quote}
Agree on the path depth issue, black out all children paths from black list.
Thanks for the review.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234333#comment-16234333
]
Jason Lowe commented on YARN-7197:
--
Thanks for updating the patch!
bq. If black list contains misconfigured item, i.e, source doesn't exist,
container creation fails.
What's the rationale behind this behavior? I could see admins configuring a
path here that may only exist on _some_ nodes but not all.
bq. If black list item is a file or a socket that already existed in docker
container image, container creation fails.
Seems like we could solve this limitation by either automatically determining
whether the blacklisted path on the host is a file or a directory or having a
way for the admin to give a directive whether the blacklisted path should be
obscured with a file or a directory.
Comments on the patch:
Rather than normalizing the banned_mounts every time, would it make more sense
to normalize them just once when we read them in from the config? That would
also remove the need to skip null entries when checking since we could weed
them out during the config read.
check_banned_mounts leaks normalized_path if banned_mounts is null.
check_banned_mounts needs to use {{strncmp}} for the length of the banned path
to catch users trying to mount beneath a blacklisted path. For example,
{{/home}} is blacklisted by the admin and the user tries to mount
{{/home/yarn}}.
How do we know mount_src is big enough to do the strcpy of /var/empty/sshd
without overrunning the buffer and scribbling on the heap?
Nit: Checking for a magic value from check_banned_mounts is odd for what is
essentially a boolean predicate function. I'd expect callers to use {{if
(banned)}} rather than knowing it's going to return {{-1}} instead of the more
natural value, {{1}}.
I think we should just fail the container launch if the user tries to mount at
or under a blacklisted path. They clearly are trying to do something the admin
does not want. We should only try to blot out blacklisted paths with an empty
file/dir if and only if the user is trying to mount above a blacklisted path
and could be legitimate behavior. Thoughts?
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
>Priority: Major
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch,
> YARN-7197.003.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226857#comment-16226857
]
Shane Kumpf commented on YARN-7197:
---
Thanks for the additional details. For that case, it is true that {{/}} would
need to be in the whitelist (which we should not allow in the white list to
enforce this). There are other API filesystems that are not top-level that this
would still apply to, but can be worked around as you said.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226800#comment-16226800
]
Jason Lowe commented on YARN-7197:
--
I was under the impression the blacklist would only mount the empty directories
if the user was attempting to mount a higher, whitelisted directory in the
blacklist path. Any attempt to mount the blacklist directly would still fail
the container as the current patch does. So in other words, a blacklist of
{{/run}} would never mount an empty directory (using a basic assumption nobody
would add {{/}} to the whitelist) and just fail the container if anyone
attempted to mount anything at {{/run}} or under it. A blacklist of
{{/run/docker.socket}} would only mount an empty file/dir onto
{{/run/docker.socket}} if {{/run}} was in the whitelist and the user tried to
mount {{/run}}.
In short, we only blacklist when we need to. If the container can't work due
to the mounted blacklisted paths then the user will need to stop mounting above
the path and instead cherry-pick the whitelisted paths around the blacklisted
ones. That's something they'll have to do anyway if we don't add this feature
and simply document how to properly setup the whitelist.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16226626#comment-16226626 ] Shane Kumpf commented on YARN-7197: --- My initial thought was that the empty directory bind mounted (Solution 3) would work nicely to achieve this goal, but I believe that could be problematic as well. A bulk of the top level directories being discussed as blacklist candidates are "API filesystems", for example /run. While containers != VMs, containers will be used similarly and as a result will run systemd. Systemd manages "API Filesystems" if they don't exist. However, in the case of an empty /run bind mount, systemd will not create the tmpfs /run, which can lead to broken behavior for units leveraging tmpfiles. This is one case I'm aware of, but I would not be surprised to find other issues on non-systemd systems. Given the discussions on how the proposed solutions may surprise users and doesn't necessarily prevent attack, I'm starting to believe documentation about the dangers of white listing top-level mounts might be most appropriate, as Jason mentioned earlier. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225812#comment-16225812 ] Jason Lowe commented on YARN-7197: -- Solution 3 is more secure since the paths are unavailable within the container. Even if there is a trojan in the container that escalates privilege, the hacker needs to break outside of the container to access the path rather than accessing it within the container directly. bq. The third solution can throw people off, if they do not know about black-list is hijacked to empty location. I'm not sure how admins are going to be confused or upset that paths in the blacklist were not what was expected. Either it shouldn't be accessed and thus shouldn't matter what's there or it needs to be there and shouldn't be in the blacklist. Or am I missing a scenario where there's an in-between? IMHO it's more confusing to users if the blacklist doesn't actually prevent access to the blacklisted paths. The point of the blacklist is that containers should not be able to access those paths on the host. bq. Mounting from parent of black list directory will depends on filesystem acl to enforce the permission. If we are simply relying on filesystem ACLs to cover us if the user mounts above the blacklist then I would argue there's no point to the blacklist. Either we trust the filesystem ACLs or we don't. If we don't then letting the admin trivially configure a setup where the user can mount above the blacklist path is not helpful or intuitive. If we are serious about not trusting the filesystem ACLs from within the container (i.e.: actually need a blacklist) then we need to do whatever we can to prevent access even if the admins (un)intentionally configure paths above the blacklisted path. That means we need to do something like Solution 3 above or fail to create the container when the user mounts above. Choosing to fail the container means we're essentially at Solution 1 where the admin has no ability to cherry-pick out paths that should not be allowed and must maintain explicit paths to things that are allowed. Then the blacklist would have no utility short of documentation of what normally should not be placed in the whitelist. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225452#comment-16225452
]
Eric Yang commented on YARN-7197:
-
[~jlowe]
{quote}Either /run isn't in the whitelist in the first place rendering the
blacklist entry moot or /run is in the whitelist and the user can simply mount
/run and access the blacklist path.{quote}
Let's expand on the real world example. A hacker tries to take control of
{{/run/docker.socket}} to acquire root privileges and spawn root containers to
access vital system area to become root on the host system. The system admin
placed {{/var}} in read-write white list for ability to write to database and
log directories, without black list capability. Hacker explicitly specify
{{/var/run/docker.socket}} to be included, put the socket in
{{/tmp/docker.socket}}. Hacker generates a docker image with /etc/group
modified to include his own name or setuid bit binary in container. Hack can
successfully gain control to host level docker without much effort.
{{/run}} contains a growing list of software that put their pid file or socket
in this location. System admin can't say no to not allow other software to
place their socket in {{/run}} location and share between containers due to
company requirement. However, he still doesn't want to let hacker gain root
access.
Solution 1:
System admin placed {{/var/*}} and {{/run/*}} (except /run/docker.socket),
carefully in read-write white list. None of the symlink is exposed. Hacker
can not get in.
Solution 2 (All symlinks are banned and explicit hardcoded locations):
(Current proposed patch)
System admin specifies:
white-list-read-write: {{/var}}, {{/run/*}} (except /run/docker.socket),
{{/mnt/hdfs/user/*}} (exception yarn)
black-list: {{/var/run}},{{/run/docker.socket}},{{/mnt/hdfs/user/yarn}}
Hacker attempt to mount a symlink location resulting in access denied from
container startup, or explicit hard coded location also result in ban.
Solution 3: (Ban symlink and replace black list location with empty directory):
(Jason proposed implementation)
System admin specifies:
white-list-read-write: {{/var}},{{/run}},{{/mnt/hdfs/user}}
black-list: {{/run/docker.socket}},{{/mnt/hdfs/user/yarn}}
Hacker attempt to mount a symlink location resulting in access denied from
container startup, or mount /run/docker.socket manually, but result in empty
file.
All solutions requires system administrator to enforce ability to upload secure
image to private registry to prevent torjan horse in docker image.
I can see the appeal that without having to do a high upkeep of
white-list-read-write directories by the new proposal. The third solution can
throw people off, if they do not know about black-list is hijacked to empty
location. However, the deeper nested directories, it would be harder to secure
by second solution. If community favors the third solution, I can revise patch
accordingly.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225397#comment-16225397 ] Eric Badger commented on YARN-7197: --- bq. I think we should fail the container creation, if black listed directory is detected. Mounting from parent of black list directory will depends on filesystem acl to enforce the permission. Having ability to substitute a subdirectory with empty directory is a feature, not a security hole, but it shouldn't be done without user's consent. If we're depending on file system acls to enforce permissions then what is the blacklist giving us? I'm still not seeing the utility of it. I guess maybe it foils script kiddies who hardcode paths into their attacks, but I don't think that's a valid reason to add in this extra configuration. If they can get to the "blacklisted" directory at all inside the container, then the feature doesn't work as a blacklist. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225228#comment-16225228
]
Eric Yang commented on YARN-7197:
-
[~ebadger] My mistakes, normal directory works. I was doing test like this:
{code}
docker run -it -v /home/:/home -v /dev/null:/home/yarn centos:7 bash
{code}
Therefore replacing black listed source mount point with {{/dev/null}} isn't a
universal solution. If we mount a empty directory from container working
directory, like Jason was suggesting, it could prevent unauthorized people from
getting into protected area.
One danger fact is that the user can make a mistake in configuration, and user
can loose all the data that was placed into the empty directory without warning.
I think we should fail the container creation, if black listed directory is
detected. Mounting from parent of black list directory will depends on
filesystem acl to enforce the permission. Having ability to substitute a
subdirectory with empty directory is a feature, not a security hole, but it
shouldn't be done without user's consent.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16225026#comment-16225026
]
Eric Badger commented on YARN-7197:
---
{quote}
Btw, docker doesn't support double mount like:
docker run -it -v /mnt/hdfs/user:/home -v /tmp/empty:/home/yarn
If /mnt/hdfs/user is already mounted, then yarn will show up.
{quote}
[~eyang], I'm seeing this to not be true in practice. If I mount /home to /home
and then /home/ebadger to /tmp/foobar, then /home contains its normal contents
except that /home/ebadger actually resolves to /tmp/foobar. I don't know how
the internals work here or if this is "supposed to" work, but this is what
happens on docker 1.12.6 on rhel7.
{noformat}
[ebadger@foobar home]$ ls -l /home/ebadger
total 1336
-rw--- 1 ebadger users 1347072 Oct 11 13:58 busy
drwxr-xr-x 2 ebadger users4096 Oct 27 19:26 conf
drwxr-xr-x 18 ebadger users4096 Oct 27 19:36 hadoop
drwxr-xr-x 4 ebadger users4096 Oct 27 20:42 hadoop-run
drwxr-xr-x 10 ebadger users4096 Oct 27 19:23 protobuf-2.5.0
drwxr-xr-x 2 ebadger users4096 Oct 27 18:27 scripts
[ebadger@foobar home]$ ls -l /home/
total 4
drwx-- 9 ebadger users 4096 Oct 30 14:03 ebadger
-rw-r--r-- 1 rootroot 0 Oct 30 14:12 user2
[ebadger@foobar home]$ ls -l /home/ebadger
total 1336
-rw--- 1 ebadger users 1347072 Oct 11 13:58 busy
drwxr-xr-x 2 ebadger users4096 Oct 27 19:26 conf
drwxr-xr-x 18 ebadger users4096 Oct 27 19:36 hadoop
drwxr-xr-x 4 ebadger users4096 Oct 27 20:42 hadoop-run
drwxr-xr-x 10 ebadger users4096 Oct 27 19:23 protobuf-2.5.0
drwxr-xr-x 2 ebadger users4096 Oct 27 18:27 scripts
[ebadger@foobar home]$ ls -l /tmp/foobar/
total 0
-rw-r--r-- 1 ebadger users 0 Oct 30 14:00 fizzgig
-rw-r--r-- 1 ebadger users 0 Oct 30 14:00 tempfile
[ebadger@foobar home]$ sudo docker run --rm -it -v /home/:/home -v
/tmp/foobar/:/home/ebadger docker.io/ubuntu:xenial /bin/bash
root@d0b86e4da455:/home# ls -l /home
total 4
drwxr-xr-x 2 x users 4096 Oct 30 14:00 ebadger
-rw-r--r-- 1 root root 0 Oct 30 14:12 user2
root@d0b86e4da455:/home# ls -l /home/ebadger/
total 0
-rw-r--r-- 1 x users 0 Oct 30 14:00 fizzgig
-rw-r--r-- 1 x users 0 Oct 30 14:00 tempfile
{noformat}
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223344#comment-16223344
]
Eric Yang commented on YARN-7197:
-
Btw, docker doesn't support double mount like:
{code}
docker run -it -v /mnt/hdfs/user:/home -v /tmp/empty:/home/yarn
{code}
If {{/mnt/hdfs/user}} is already mounted, then yarn will show up. We have to
trust that yarn directory permission is setup correctly. Bind mount a
directory with blank directory will not solve any problem to hide /home/yarn
directory. Filesystem ACL is the only safe guard. It's not possible to make a
subdirectory disappear.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223306#comment-16223306
]
Eric Yang commented on YARN-7197:
-
[~jlowe] I agree with everything you said that the current implementation is a
moot point besides top level directories. The real security problem with mount
was solved in YARN-4266, where enforcing effective group will ensure the
processes inside the container doesn't get more privileges than the user was
allowed. The intend of black list is supposed to add additional safe guard to
prevent mistakes that can result in jail break. If we go by what you said,
then it would be {{/mnt/hdfs/user/yarn}} in black list, it automatically gets
replaced with {{/dev/null}}.
If someone make an attack like:
{code}
docker run -v /mnt/hdfs/user/yarn:/tmp/yarn centos:latest bash
{code}
The resulting command looks like:
{code}
docker run -v /dev/null:/tmp/yarn centos:latest bash
{code}
Inside the container looks like:
{code}
[root@6d6588764109 /]# ls -l /tmp
total 4
-rwx-- 1 root root 836 Sep 11 15:53 ks-script-q6TWGF
crw-rw-rw- 1 root root 1, 3 Oct 28 07:37 yarn
-rw--- 1 root root0 Sep 11 15:51 yum.log
{code}
Everything goes no where, and no clean up. Thoughts?
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16222610#comment-16222610
]
Jason Lowe commented on YARN-7197:
--
Like [~ebadger], I am a bit confused on how this adds a lot of value. The only
way I see this being useful is if the blacklist consists of top-level
directories like {{/run}} and {{/etc}}. Otherwise any blacklist path that
isn't a top-level directory is probably going to fool some users into thinking
the blacklist support really does prevent the container from ever getting
access to the path. But that's not going to be the case if a parent directory
of the blacklist path is in the whitelist because the container can simply
mount above the blacklist path and still see it. And if the blacklist path
parents are not in the whitelist then why do we need the blacklist entry?
I'm still wondering what a real-world use-case for this is as it works in the
patch. The closest I've seen is [~shaneku...@gmail.com]'s
[comment|https://issues.apache.org/jira/browse/YARN-7197?focusedCommentId=16218724=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16218724]
stating we would put some top-level paths in the blacklist and they are only
there to make a sysadmin think twice about adding something under those paths
to the whitelist. I guess that's OK, but I don't see how the blacklist adds
any value if it contains a path that isn't a top-level path.
bq. In the examples above, allowing people to read yarn system directory can
leak credentials about other users. Allowing user to mount /run/docker.socket
can let user jail break docker container to become root.
Those very same examples are why I don't see the value. IIUC these two
examples are from [this
comment|https://issues.apache.org/jira/browse/YARN-7197?focusedCommentId=16217854=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16217854].
For the first one, /mnt/hdfs/user is in the whitelist so a nefarious user can
simply mount the entire /mnt/hdfs/user hierarchy and then can access the yarn
files as Eric said. I don't see how a blacklist of /mnt/hdfs/user/yarn buys
any value at all here unless we're assuming the nefarious user isn't clever.
Similarly I don't see the point of blacklisting /run/docker.socket. Either
/run isn't in the whitelist in the first place rendering the blacklist entry
moot or /run is in the whitelist and the user can simply mount /run and access
the blacklist path. What is a use-case where the a blacklist path isn't a
top-level directory and it actually helps prevent an attack from a mildly
clever nefarious user? And if it's a top-level directory we're arguably
already covered because those paths are not in the whitelist, so this ends up
as documentation of paths we really don't want admins to configure unless they
know what's going on rather than a useful tool for admins to use for their own
paths.
If we really want to prevent access to the blacklist paths couldn't we
bind-mount a read-only empty directory or something similarly useless to the
blacklist paths? The paths will still appear in the container but won't expose
anything. Bonus points for only doing the blacklist mounts that are necessary
relative to the other mounts. Then I can see how this becomes a useful tool
for admins. The admin can allow /mnt/hdfs/user in the whitelist but users will
not be able to see anything under /mnt/hdfs/user/yarn even if they mount all of
/mnt/hdfs/user. And now the blacklist really does black out the listed paths
no matter what the user tries to do or what the whitelist contains.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16221402#comment-16221402 ] Eric Yang commented on YARN-7197: - [~ebadger] Any other concern about this feature? I will commit this, if there is no other objection. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16219115#comment-16219115
]
Eric Yang commented on YARN-7197:
-
[~ebadger] said:
{quote}
The user can just mount above the blacklist and they get access to exactly what
they want. This protects them from mounting the exact path in the blacklist,
but that doesn't really buy us anything if they can mount the parent directory.
If I can't prevent a file/directory underneath the parent directory from being
accessed, then I don't see the utility of the blacklist.
{quote}
Black list is not the inverse of white list in this context. Black list is
designed to prevent certain exact path to be mounted. Such as /dev, /proc,
/sys, and /run. In the examples above, allowing people to read yarn system
directory can leak credentials about other users. Allowing user to mount
/run/docker.socket can let user jail break docker container to become root.
Black list can prevent system api from being mounted to minimize attack
surface. Paranoid admin might configure docker to use socket path other than
/run/docker.socket, and put the customized location in black list. The same
applies to YARN system directories. Black list increases degree of difficulty
for host to be cracked by keeping programmable API away from inside the
containers.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218724#comment-16218724 ] Shane Kumpf commented on YARN-7197: --- The way I saw this being useful was less about security/blocking access and more about making an admin consider the consequences before allowing mounts that are very likely to negatively impact the host; e.g. /run or /etc. It's a way for the contributors to say "you'll probably shoot yourself in the foot in you allow this mount, we don't recommend you do, but if you really want to change the blacklist, you're the admin and can do what you want." While mounts in here won't apply to all container types/use cases, it would force the admin to pause and, hopefully, figure out why that mount is in the blacklist. For example, If you mount /run r/w in a container running systemd, and the host is also running systemd, there is a very good chance you can no longer log in to the host system until a reboot happens. Let's hope I didn't just ask for 100k of these containers and kill my whole cluster. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16218663#comment-16218663
]
Eric Badger commented on YARN-7197:
---
bq. File system ACL is the only protection to verify uid:gid are indeed
authorized to access the included area.
If that's true, then I don't see what the black list gives us
{quote}
The black list is designed to filter out more sinister attack against the
system.
For example, system admin configures:
white-listed-read-write: /mnt/hdfs/user
black-listed: /mnt/hdfs/user/yarn,/run/docker.socket
This will prevent naughty junior developer from doing:
docker run -u yarn:yarn -it -v /mnt/hdfs/user/yarn:/tmp centos:latest bash
or
docker run -u yarn:docker -it -v /run/docker.socket:/run/docker.socket
centos:latest bash
{quote}
But there's nothing preventing the attacker from running
{noformat}
docker run -u yarn:yarn -it -v /mnt/hdfs/user:/tmp centos:latest bash
{noformat}
and then using /tmp/yarn instead of /tmp to get to /mnt/hdfs/user/yarn. Same
applies in the /run/docker.socket if /run were in the whitelist
bq. The black list feature is not designed to make a subdirectory disappear.
Docker still depends on file system acl to enforce security. This feature is
only good for blocking a certain system directories from developers to protect
host OS and Hadoop. This is also the reason that system admin keeps black list
secrets from naughty developers.
But I don't see it blocking directories at all. The user can just mount above
the blacklist and they get access to exactly what they want. This protects them
from mounting the exact path in the blacklist, but that doesn't really buy us
anything if they can mount the parent directory. If I can't prevent a
file/directory underneath the parent directory from being accessed, then I
don't see the utility of the blacklist.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217854#comment-16217854
]
Eric Yang commented on YARN-7197:
-
[~ebadger] You are correct on all points, and mounting parent directory will
allow container to include directories that are supposedly in the banned list.
File system ACL is the only protection to verify uid:gid are indeed authorized
to access the included area.
The black list is designed to filter out more sinister attack against the
system.
For example, system admin configures:
{code}
white-listed-read-write: /mnt/hdfs/user
black-listed: /mnt/hdfs/user/yarn
{code}
This will prevent aughty junior developer from doing:
{code}
docker run -u yarn:yarn -it -v /mnt/hdfs/user/yarn:/tmp centos:latest bash
{code}
and this works:
{code}
docker run -u 501:501 -it -v /mnt/hdfs/user:/home centos:latest bash
{code}
The black list feature is not designed to make a subdirectory disappear.
Docker still depends on file system acl to enforce security. This feature is
only good for blocking a certain system directories from developers to protect
host OS and Hadoop. This is also the reason that system admin keep black list
secrets from naughty developers.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217795#comment-16217795
]
Eric Badger commented on YARN-7197:
---
Actually, thinking about this more, I'm wondering if this blacklist is going to
work at all in the way that has been proposed here. The aim, as I see it, is to
allow fairly permissive access at a high level and then cherry-pick things to
disallow access to beneath it. However, isn't this fundamentally broken since
we can just bind mount the higher level directory? Taking your example from
above
bq. For example, if a system admin would like to allow mounting of
/mnt/hdfs/user/${USER} but not /mnt/hdfs/user/yarn using nfs gateway.
If we allow /mnt/hdfs/user in the whitelist, then we can mount that directory
to get both /mnt/hdfs/user/${USER} and /mnt/hdfs/user/yarn, which will be
underneath it. Adding /mnt/hdfs/user/yarn to the blacklist won't prevent us
from mounting /mnt/hdfs/user, which will then include /mnt/hdfs/user/yarn.
Unless there's a way to hide underlying files/directories in docker, I don't
see how this method can work. Please let me know if I'm mistaken and/or
misunderstanding something here.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217693#comment-16217693
]
Eric Badger commented on YARN-7197:
---
[~eyang]
bq. Symlink are banned to prevent mistakes. I can lift the symlink check and
trust that system admin will always express physical path correctly instead of
symlink paths in black list
I think that if we're going to allow symlinks in the whitelist case then we
should allow them here as well. Also, allowing symlinks in the white/black
lists can be useful. For example, if you have /hadoop symlinked to the most
recent hadoop version, maybe you want to add that to the whitelist, but you
don't want to change the whitelist everytime you update your hadoop version.
{quote}
I did not separate black list for read-only and read-write to reduce wordy
configurations that could generate security holes. For example, if someone
would like to mount /run as read only, and allow docket.socket to be used
inside container.
Should they specify?
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
black-listed-read-only:/run/docker.socket
black-listed-read-write: /run
Or?
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
black-listed-read-only: /run/docker.socket
Ideally, user should be able to specify the minimum to get the system to work:
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
or
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
black-listed: /var/run
{quote}
Even with the blacklist split up into ro and rw, why wouldn't they be able to
simply add what they want to the whitelist and not bother with the blacklist?
Splitting the blacklist gives you more fine-grained control over exactly what
you want to disallow. For example, if you had a directory where you wanted to
be able to mount almost everything as rw, except for a few things. Then you'd
whitelist the whole directory as rw and use the blacklist to disallow the
specific directories as ro/rw. Maybe you have a keytab under one of those
directories that you need to disallow rw access and you have a conf directory
that you only want to be read, not written. If you don't split up the blacklist
then you would lose the ro/rw control on those files and they would have to be
either all or nothing.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217638#comment-16217638
]
Eric Yang commented on YARN-7197:
-
[~ebadger] Symlink are banned to prevent mistakes. I can lift the symlink
check and trust that system admin will always express physical path correctly
instead of symlink paths in black list. I found myself expressing
/var/run/utmp more often instead of /run/utmp, and double negative could slip
through undetected when symlinks are supported for black list.
I did not separate black list for read-only and read-write to reduce wordy
configurations that could generate security holes. For example, if someone
would like to mount /run as read only, and allow docket.socket to be used
inside container.
Should they specify?
{code}
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
black-listed-read-only:/run/docker.socket
black-listed-read-write: /run
{code}
Or?
{code}
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
black-listed-read-only:/run/docker.socket
{code}
Ideally, user should be able to specify the minimum to get the system to work:
{code}
white-listed-read-only: /run
white-listed-read-write: /run/docker.socket
{code}
or
{code}
white-listed-read-only: /run
black-listed-read-only: /run/docker.socket
{code}
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217393#comment-16217393
]
Eric Badger commented on YARN-7197:
---
{noformat}
// symlinks are banned.
if (strcmp(normalized_path, requested) != 0) {
return -1;
}
{noformat}
Why are we outright banning symlinks? We allow them in the whitelist, so I
don't see why this should be any different.
{noformat}
permitted_rw |= check_banned_mounts((const char **) banned_mounts,
mount_src);
permitted_ro |= check_banned_mounts((const char **) banned_mounts,
mount_src);
{noformat}
This looks a little weird since permitted_rw and permitted_ro are ints. I guess
it works since (1 |= -1) == -1, but it might lead to confusion down the road.
It would also be nice to know whether the mount was blocked via the whitelist
or the blacklist.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Eric Yang
> Attachments: YARN-7197.001.patch, YARN-7197.002.patch
>
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217162#comment-16217162 ] Eric Badger commented on YARN-7197: --- [~eyang], since we split the whitelist into separate ro and rw lists, should we do the same here? > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216245#comment-16216245 ] Eric Yang commented on YARN-7197: - The failed unit test is not related to this patch. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch, YARN-7197.002.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215619#comment-16215619
]
Hadoop QA commented on YARN-7197:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
32m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 42s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 51s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:ca8ddc6 |
| JIRA Issue | YARN-7197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12893546/YARN-7197.002.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux f7a35f51d058 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 921338c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_131 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18088/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18088/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18088/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208479#comment-16208479 ] Eric Badger commented on YARN-7197: --- [~eyang], just like YARN-5534, I'll hold off looking at this patch until YARN-7344 is resolved > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16206912#comment-16206912
]
Hadoop QA commented on YARN-7197:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 16s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 59s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
15s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7197 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892502/YARN-7197.001.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux a7fb3ca13e1b 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b406d8e |
| Default Java | 1.8.0_144 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17978/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17978/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17978/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL:
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167890#comment-16167890
]
Eric Badger commented on YARN-7197:
---
{quote}
For example, if a system admin would like to allow mounting of /mnt/hdfs/user/$
\{USER\}
but not /mnt/hdfs/user/yarn using nfs gateway.
{quote}
Ok that makes sense from a usability point of view. I think this should be
identified in a comment/in the docs so that it's clear what the intentions of
both the whitelist and blacklist are. Most situations should be handled via the
whitelist and won't have to touch the blacklist at all
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167752#comment-16167752 ] Shane Kumpf commented on YARN-7197: --- [~ebadger] - Thanks for the feedback. I had similar concerns, but after having this occur in a real world scenario, I believe there is value. This would help make sure an admin has to think twice before enabling a known dangerous mount. I'm confident a lazy admin will whitelist / and come complain when the system gets broken beyond all repair. :) > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[
https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16167123#comment-16167123
]
Eric Yang commented on YARN-7197:
-
[~ebadger] This is similar to firewall rules. We need a way to define ACCEPTED
and REJECTED rules. White list could be cumbersome to express an exhaustive
list of mount points sometimes. If there are a lot of paths intermixed by a lot
of allow list, and few of black list path.
For example, if a system admin would like to allow mounting of
/mnt/hdfs/user/${USER} but not /mnt/hdfs/user/yarn using nfs gateway. White
list can not express the proper constraints alone. If white list has some
smarts to define if condition, black list might not be necessary.
> Add support for a volume blacklist for docker containers
>
>
> Key: YARN-7197
> URL: https://issues.apache.org/jira/browse/YARN-7197
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>
> Docker supports bind mounting host directories into containers. Work is
> underway to allow admins to configure a whilelist of volume mounts. While
> this is a much needed and useful feature, it opens the door for
> misconfiguration that may lead to users being able to compromise or crash the
> system.
> One example would be allowing users to mount /run from a host running
> systemd, and then running systemd in that container, rendering the host
> mostly unusable.
> This issue is to add support for a default blacklist. The default blacklist
> would be where we put files and directories that if mounted into a container,
> are likely to have negative consequences. Users are encouraged not to remove
> items from the default blacklist, but may do so if necessary.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166937#comment-16166937 ] Eric Badger commented on YARN-7197: --- Hi [~shaneku...@gmail.com], [~eyang]. I'm not sure I see the usefulness here given the proposed whitelist in YARN-5534. Whitelists are more secure by default whereas blacklists need to be configured to be secure and have to constantly change. It makes sense to me to have a whitelist, but I don't see the reason behind having a separate blacklist. If the admin wanted to allow a potentially dangerous directory to be bind-mounted, they would have to explicitly add it. If anything, this proposed blacklist seems like it should be a comment on the whitelist warning the user against adding these specific dangerous directories to the list. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166916#comment-16166916 ] Eric Yang commented on YARN-7197: - Consider the following scenarios: # Docker container with --privileged=true flag enabled and run systemd in docker container with: /sys/fs/cgroup:/sys/fs/cgroup:ro. This allows systemd to run in docker container, while protecting cgroup control enforced from host layer. # Docker container attempt to mount /run should be forbidden, always initialized with tmpfs. # Allow docker container to access /var/run/docker.socket on the host layer for privileged container to interact with host layer docker daemon. The black list needs to have ability to list all the forbidden mount points, such as /sys, and /run. There are some exception that needs to have ability to mount as read only. The last but not the least, the ability to create special mount points for privileged container to access host layer docker daemon. This feature requires to maintain 3 control lists for accuracy. # A general backlisted mount points # A read-only mount points # A exception list that is allowed to mount, if running with privileged mode. Last one can be covered by white list feature in YARN-5534. This JIRA only needs to cover case 1, and 2. > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
