[
https://issues.apache.org/jira/browse/YARN-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16910839#comment-16910839
]
Eric Yang commented on YARN-9564:
---------------------------------
[~ebadger] what is the right procedure for building image locally and upload to
hdfs?
Tried various approach, and none seems to work:
to push-update from local docker image to hdfs:
{code}
docker images
REPOSITORY TAG IMAGE ID
CREATED SIZE
runc-centos latest 467de23cbfb5
16 minutes ago 545MB
local/runc-centos latest 467de23cbfb5
16 minutes ago 545MB
{code}
Pushing image to hdfs result in errors:
{code}
# ./docker_to_squash.py --hadoop-prefix=/usr/local/hadoop-3.3.0-SNAPSHOT
--log=DEBUG pull-build-push-update
local/runc-centos:latest,local/runc-centos:latest
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'version']
DEBUG: command: ['skopeo', '-v']
DEBUG: command: ['mksquashfs', '-version']
DEBUG: args: Namespace(LOG_LEVEL='DEBUG', check_magic_file=False, force=False,
func=<function pull_build_push_update at 0x13c7a28>,
hadoop_prefix='/usr/local/hadoop-3.3.0-SNAPSHOT', hdfs_root='/runc-root',
image_tag_to_hash='image-tag-to-hash',
images_and_tags=['local/runc-centos:latest,local/runc-centos:latest'],
magic_file='etc/dockerfile-version', pull_format='docker', replication=1,
skopeo_format='dir', sub_command='pull-build-push-update',
working_dir='dts-work-dir')
DEBUG: extra: []
DEBUG: image-tag-to-hash: image-tag-to-hash
DEBUG: LOG_LEVEL: DEBUG
DEBUG: HADOOP_BIN_DIR: /usr/local/hadoop-3.3.0-SNAPSHOT/bin
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root']
DEBUG: Setting up squashfs dirs: ['/runc-root/layers', '/runc-root/config',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/layers']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/config']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-chmod',
'755', '/runc-root/manifests']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-ls',
'/runc-root/image-tag-to-hash']
DEBUG: command: ['/usr/local/hadoop-3.3.0-SNAPSHOT/bin/hadoop', 'fs', '-get',
'/runc-root/image-tag-to-hash', 'dts-work-dir/image-tag-to-hash']
2019-08-19 23:35:55,405 INFO sasl.SaslDataTransferClient: SASL encryption trust
check: localHostTrusted = false, remoteHostTrusted = false
INFO: Working on image local/runc-centos:latest with tags
['local/runc-centos:latest']
DEBUG: command: ['sudo', 'skopeo', 'inspect', '--raw',
'docker://local/runc-centos:latest']
FATA[0001] Error reading manifest latest in docker.io/local/runc-centos: errors:
denied: requested access to the resource is denied
unauthorized: authentication required
DEBUG: command: ['sudo', 'rm', '-rf', 'dts-work-dir']
Traceback (most recent call last):
File "./docker_to_squash.py", line 1283, in <module>
main()
File "./docker_to_squash.py", line 1280, in main
args.func(args)
File "./docker_to_squash.py", line 560, in pull_build_push_update
manifest, manifest_hash = get_manifest_from_docker_image(pull_format, image)
File "./docker_to_squash.py", line 237, in get_manifest_from_docker_image
False, True, True)
File "./docker_to_squash.py", line 67, in shell_command
+ str(out) + "\nstderr: " + str(err))
Exception: Commmand: ['sudo', 'skopeo', 'inspect', '--raw',
'docker://local/runc-centos:latest'] failed with returncode: 1
stdout:
stderr: None
{code}
> Create docker-to-squash tool for image conversion
> -------------------------------------------------
>
> Key: YARN-9564
> URL: https://issues.apache.org/jira/browse/YARN-9564
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Badger
> Assignee: Eric Badger
> Priority: Major
> Attachments: YARN-9564.001.patch, YARN-9564.002.patch,
> YARN-9564.003.patch
>
>
> The new runc runtime uses docker images that are converted into multiple
> squashfs images. Each layer of the docker image will get its own squashfs
> image. We need a tool to help automate the creation of these squashfs images
> when all we have is a docker image
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
