[jira] [Comment Edited] (YARN-9184) Docker run doesn't pull down latest image if the image exists locally
[ https://issues.apache.org/jira/browse/YARN-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16763883#comment-16763883 ] Eric Yang edited comment on YARN-9184 at 2/8/19 9:14 PM: - [~ebadger] {quote}This isn't necessarily true. You can query the docker registry to get the hash of any arbitrary image and tag.{quote} Credential to access docker is stored in ~/.docker/config.json. We need to aggregate docker configuration to find out protocol, server, port, username, and secret to login to the remote docker registry. Do we want to let container-executor venture outside of working directory to aggregation those information? It seems risky. [~billie.rinaldi] Pull digest does not guarantee the retrieved image has same name as user specified image name because the latest tag on repository may have moved to another target. I don't see a way for container-executor ACL to match pulled digest images. We should take job image consistency to a separate JIRA from this one. YARN-9292 filed to track image consistency per application. was (Author: eyang): [~ebadger] {quote}This isn't necessarily true. You can query the docker registry to get the hash of any arbitrary image and tag.{quote} Credential to access docker is stored in ~/.docker/config.json. We need to aggregate docker configuration to find out protocol, server, port, username, and secret to login to the remote docker registry. Do we want to let container-executor venture outside of working directory to aggregation those information? It seems risky. [~billie.rinaldi] Pull digest does not guarantee the retrieved image has same name as user specified image name because the latest tag on repository may have moved to another target. I don't see a way for container-executor ACL to match pulled digest images. We should take job image consistency to a separate JIRA from this one. > Docker run doesn't pull down latest image if the image exists locally > -- > > Key: YARN-9184 > URL: https://issues.apache.org/jira/browse/YARN-9184 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.1.0, 3.0.3 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9184.001.patch, YARN-9184.002.patch, > YARN-9184.003.patch, YARN-9184.004.patch > > > See [docker run doesn't pull down latest image if the image exists > locally|https://github.com/moby/moby/issues/13331]. > So, I think we should pull image before run to make image always latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9184) Docker run doesn't pull down latest image if the image exists locally
[ https://issues.apache.org/jira/browse/YARN-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758170#comment-16758170 ] Zhaohui Xin edited comment on YARN-9184 at 2/1/19 10:14 AM: [~liushaohui], Thanks for your comment. If this config is in job side, it will bring some problems. You can read the comments above for more details.:D {quote}The images could be significantly different if an image was pushed halfway through a job. Then some containers would run with the old image and the rest would run with the newer image. {quote} was (Author: uranus): [~liushaohui], Hi, Thanks for your comment. If this config is in job side, it will bring some problems. You can read the comments above for more details.:D {quote}The images could be significantly different if an image was pushed halfway through a job. Then some containers would run with the old image and the rest would run with the newer image. {quote} > Docker run doesn't pull down latest image if the image exists locally > -- > > Key: YARN-9184 > URL: https://issues.apache.org/jira/browse/YARN-9184 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.1.0, 3.0.3 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9184.001.patch, YARN-9184.002.patch, > YARN-9184.003.patch, YARN-9184.004.patch > > > See [docker run doesn't pull down latest image if the image exists > locally|https://github.com/moby/moby/issues/13331]. > So, I think we should pull image before run to make image always latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9184) Docker run doesn't pull down latest image if the image exists locally
[ https://issues.apache.org/jira/browse/YARN-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16758134#comment-16758134 ] Liu Shaohui edited comment on YARN-9184 at 2/1/19 9:24 AM: --- Nice feature. It will be better if this option can be set per job, Eg: setting DOCKER_PULL_POLICY=always in container env. was (Author: liushaohui): Nice feature. It will better if this option can be set per job. Eg: setting DOCKER_PULL_POLICY=always in container env > Docker run doesn't pull down latest image if the image exists locally > -- > > Key: YARN-9184 > URL: https://issues.apache.org/jira/browse/YARN-9184 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.1.0, 3.0.3 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9184.001.patch, YARN-9184.002.patch, > YARN-9184.003.patch, YARN-9184.004.patch > > > See [docker run doesn't pull down latest image if the image exists > locally|https://github.com/moby/moby/issues/13331]. > So, I think we should pull image before run to make image always latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9184) Docker run doesn't pull down latest image if the image exists locally
[ https://issues.apache.org/jira/browse/YARN-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16739968#comment-16739968 ] Zhaohui Xin edited comment on YARN-9184 at 1/11/19 2:49 AM: Hi [~ebadger], thanks for the review. I added new patch. Move containerIdStr to above just because _pullImageFromRemote_ function need it for logging. {code:java} pullImageFromRemote(String containerIdStr, String imageName){code} was (Author: uranus): Hi [~ebadger], thanks for the review. I added new patch. Move containerIdStr to above just because _pullImageFromRemote_ function __ need it for logging. {code:java} pullImageFromRemote(String containerIdStr, String imageName){code} > Docker run doesn't pull down latest image if the image exists locally > -- > > Key: YARN-9184 > URL: https://issues.apache.org/jira/browse/YARN-9184 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.1.0, 3.0.3 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9184.001.patch, YARN-9184.002.patch > > > See [docker run doesn't pull down latest image if the image exists > locally|https://github.com/moby/moby/issues/13331]. > So, I think we should pull image before run to make image always latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9184) Docker run doesn't pull down latest image if the image exists locally
[ https://issues.apache.org/jira/browse/YARN-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738960#comment-16738960 ] Zhaohui Xin edited comment on YARN-9184 at 1/10/19 3:42 AM: Hi, [~ebadger], very goog suggestions. I attached patch, can you help me review this? :D was (Author: uranus): [~ebadger], very goog suggestions. I attached patch, can you help me review this? :D > Docker run doesn't pull down latest image if the image exists locally > -- > > Key: YARN-9184 > URL: https://issues.apache.org/jira/browse/YARN-9184 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.1.0, 3.0.3 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9184.001.patch > > > See [docker run doesn't pull down latest image if the image exists > locally|https://github.com/moby/moby/issues/13331]. > So, I think we should pull image before run to make image always latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9184) Docker run doesn't pull down latest image if the image exists locally
[ https://issues.apache.org/jira/browse/YARN-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16738960#comment-16738960 ] Zhaohui Xin edited comment on YARN-9184 at 1/10/19 3:42 AM: Hi, [~ebadger], very good suggestions. I attached patch, can you help me review this? :D was (Author: uranus): Hi, [~ebadger], very goog suggestions. I attached patch, can you help me review this? :D > Docker run doesn't pull down latest image if the image exists locally > -- > > Key: YARN-9184 > URL: https://issues.apache.org/jira/browse/YARN-9184 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.1.0, 3.0.3 >Reporter: Zhaohui Xin >Assignee: Zhaohui Xin >Priority: Major > Attachments: YARN-9184.001.patch > > > See [docker run doesn't pull down latest image if the image exists > locally|https://github.com/moby/moby/issues/13331]. > So, I think we should pull image before run to make image always latest. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org