[
https://issues.apache.org/jira/browse/YARN-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013200#comment-17013200
]
Eric Yang commented on YARN-9292:
---------------------------------
{quote}I see. Why isn't the AM run inside of a Docker container?{quote}
Very good question, and the answer is somewhat complicated. For AM to run in
the docker container, AM must have identical Hadoop client bits (Java, Hadoop,
etc), and credential mapping (nscd/sssd). Many of those pieces can not be
moved cleanly into Docker container in the first implementation of YARN native
service (LLAP/Slider alike projects) because resistance of building agreeable
docker image as part of Hadoop project. AM remains as outside of docker
container for simplicity.
{quote}The node might have an old Docker image on it. It would be nice to get
the image information from the registry and only fall back to the local node's
version if the registry lookup fails. An indirect way to do this would be to do
a docker pull} before calling {{docker images.{quote}
The same can be argued for people who do not want to automatic pulling of
docker image to latest. As the result, there is a flag implemented in
YARN-9184. The flag decides if it will be base on local latest or repository
latest. This change should work in combination with YARN-9184.
{quote}If we can hit the docker registry directly via its REST API then we
won't need to invoke the container-executor at all and we can avoid this
problem. This looks like it should be fairly trivial, but I don't know how much
more difficult secure registries would be.{quote}
We don't contact docker registry directly nor we have code to conect secure
docker registry. I think it is too risky to contact the registry directly
because the registry could be a private registry defined in user's docker
config.json. It would be going down a rabbit hole to follow this path.
{quote}Do you have documentation handy for docker image inspect that talks
about the fuzzy matching?{quote}
Sorry, don't have any, but here is a example that you can try locally:
{code}$ docker images|grep centos
centos 7 9f38484d220f 10 months
ago 202MB
centos latest 9f38484d220f 10 months
ago 202MB{code}
Suppose that you have used both centos:7 and centos:latest tags, and that are
both pointed to the same image. The result of the repository digest hash for
both images produces different hash:
{code}$ docker inspect image centos -f "{{.RepoDigests}}"
[centos@sha256:a799dd8a2ded4a83484bbae769d97655392b3f86533ceb7dd96bbac929809f3c
centos@sha256:b5e66c4651870a1ad435cd75922fe2cb943c9e973a9673822d1414824a1d0475]{code}
By using either hash is fine, they will result in the same image. It is
somewhat fuzzy because they are alias of one another.
{quote}A space before and after != and ==. If the purpose of omitting the
spaces is to show operation bundling, then I would just add () around the two
separate comparisons around the &&{quote}
I see, will update according. Thanks
{quote}Thanks for clearing up the quoting issue. But I'm still getting what
appears to be a less than ideal result. Is this expected behavior?{quote}
Maybe need to upgrade the docker version. The output appears like this on my
system:
{code}$ docker images --format="{{json .}}" --filter="dangling=false"
{"Containers":"N/A","CreatedAt":"2019-11-11 11:23:08 -0500
EST","CreatedSince":"2 months
ago","Digest":"\u003cnone\u003e","ID":"7317640d555e","Repository":"prom/prometheus","SharedSize":"N/A","Size":"130MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"130.2MB"}
{"Containers":"N/A","CreatedAt":"2019-07-15 16:14:12 -0400
EDT","CreatedSince":"5 months
ago","Digest":"\u003cnone\u003e","ID":"771e0613a264","Repository":"ozonesecure_kdc","SharedSize":"N/A","Size":"127MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"127.4MB"}
{"Containers":"N/A","CreatedAt":"2019-07-15 00:04:39 -0400
EDT","CreatedSince":"5 months
ago","Digest":"\u003cnone\u003e","ID":"48b0eebc96f0","Repository":"jaegertracing/all-in-one","SharedSize":"N/A","Size":"48.7MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"48.71MB"}
{"Containers":"N/A","CreatedAt":"2019-07-02 14:56:10 -0400
EDT","CreatedSince":"6 months
ago","Digest":"\u003cnone\u003e","ID":"f38d9c7e49be","Repository":"flokkr/hadoop","SharedSize":"N/A","Size":"503MB","Tag":"2.7.7","UniqueSize":"N/A","VirtualSize":"503.3MB"}
{"Containers":"N/A","CreatedAt":"2019-06-25 14:27:08 -0400
EDT","CreatedSince":"6 months
ago","Digest":"\u003cnone\u003e","ID":"c912f3f026ed","Repository":"grafana/grafana","SharedSize":"N/A","Size":"249MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"248.5MB"}
{"Containers":"N/A","CreatedAt":"2019-06-24 19:37:36 -0400
EDT","CreatedSince":"6 months
ago","Digest":"\u003cnone\u003e","ID":"1918121ff825","Repository":"vimagick/iptables","SharedSize":"N/A","Size":"10.8MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"10.76MB"}
{"Containers":"N/A","CreatedAt":"2019-06-17 11:34:25 -0400
EDT","CreatedSince":"6 months
ago","Digest":"\u003cnone\u003e","ID":"6b7f4580981c","Repository":"apache/ozone-runner","SharedSize":"N/A","Size":"721MB","Tag":"20190617-2","UniqueSize":"N/A","VirtualSize":"721.1MB"}
{"Containers":"N/A","CreatedAt":"2019-03-14 17:19:53 -0400
EDT","CreatedSince":"10 months
ago","Digest":"\u003cnone\u003e","ID":"9f38484d220f","Repository":"centos","SharedSize":"N/A","Size":"202MB","Tag":"7","UniqueSize":"N/A","VirtualSize":"201.8MB"}
{"Containers":"N/A","CreatedAt":"2019-03-14 17:19:53 -0400
EDT","CreatedSince":"10 months
ago","Digest":"\u003cnone\u003e","ID":"9f38484d220f","Repository":"centos","SharedSize":"N/A","Size":"202MB","Tag":"latest","UniqueSize":"N/A","VirtualSize":"201.8MB"}
{"Containers":"N/A","CreatedAt":"2019-03-07 21:13:34 -0500
EST","CreatedSince":"10 months
ago","Digest":"\u003cnone\u003e","ID":"e9ea51023687","Repository":"openjdk","SharedSize":"N/A","Size":"105MB","Tag":"8u191-jdk-alpine3.9","UniqueSize":"N/A","VirtualSize":"104.9MB"}
[eyang@localhost hadoop]$ docker -v
Docker version 18.09.5, build e8ff056{code}
{quote}Another question I have is around NMWebServices. Is this completely wide
open to anyone who can craft the correct endpoint or is there authentication
done before the docker images command will be run?{quote}
No, this rest api is secured by SPNEGO authentication, same as rest of node
manager rest api. HttpUtil.connect handles Kerberos negotiation.
> Implement logic to keep docker image consistent in application that uses
> :latest tag
> ------------------------------------------------------------------------------------
>
> Key: YARN-9292
> URL: https://issues.apache.org/jira/browse/YARN-9292
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN-9292.001.patch, YARN-9292.002.patch,
> YARN-9292.003.patch, YARN-9292.004.patch, YARN-9292.005.patch,
> YARN-9292.006.patch, YARN-9292.007.patch
>
>
> Docker image with latest tag can run in YARN cluster without any validation
> in node managers. If a image with latest tag is changed during containers
> launch. It might produce inconsistent results between nodes. This is surfaced
> toward end of development for YARN-9184 to keep docker image consistent
> within a job. One of the ideas to keep :latest tag consistent for a job, is
> to use docker image command to figure out the image id and use image id to
> propagate to rest of the container requests. There are some challenges to
> overcome:
> # The latest tag does not exist on the node where first container starts.
> The first container will need to download the latest image, and find image
> ID. This can introduce lag time for other containers to start.
> # If image id is used to start other container, container-executor may have
> problems to check if the image is coming from a trusted source. Both image
> name and ID must be supply through .cmd file to container-executor. However,
> hacker can supply incorrect image id and defeat container-executor security
> checks.
> If we can over come those challenges, it maybe possible to keep docker image
> consistent with one application.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
