[jira] [Commented] (MESOS-6653) Overlayfs backend may fail to mount the rootfs if both container image and image volume are specified.
[
https://issues.apache.org/jira/browse/MESOS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856464#comment-15856464
]
Alexander Rukletsov commented on MESOS-6653:
[~gilbert], [~jieyu], [~tillt] I'm retargeting this for 1.1.2. Please either
backport it to 1.1.x or remove the target version.
> Overlayfs backend may fail to mount the rootfs if both container image and
> image volume are specified.
> --
>
> Key: MESOS-6653
> URL: https://issues.apache.org/jira/browse/MESOS-6653
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Gilbert Song
>Assignee: Gilbert Song
>Priority: Critical
> Labels: backend, containerizer, overlayfs
> Fix For: 1.2.0
>
>
> Depending on MESOS-6000, we use symlink to shorten the overlayfs mounting
> arguments. However, if more than one image need to be provisioned (e.g., a
> container image is specified while image volumes are specified for the same
> container), the symlink .../backends/overlay/links would fail to be created
> since it exists already.
> Here is a simple log when we hard code overlayfs as our default backend:
> {noformat}
> [07:02:45] : [Step 10/10] [ RUN ]
> Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0
> [07:02:46] : [Step 10/10] I1127 07:02:46.416021 2919
> containerizer.cpp:207] Using isolation:
> filesystem/linux,volume/image,docker/runtime,network/cni
> [07:02:46] : [Step 10/10] I1127 07:02:46.419312 2919
> linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy
> for the Linux launcher
> [07:02:46] : [Step 10/10] E1127 07:02:46.425336 2919 shell.hpp:107]
> Command 'hadoop version 2>&1' failed; this is the output:
> [07:02:46] : [Step 10/10] sh: 1: hadoop: not found
> [07:02:46] : [Step 10/10] I1127 07:02:46.425379 2919 fetcher.cpp:69]
> Skipping URI fetcher plugin 'hadoop' as it could not be created: Failed to
> create HDFS client: Failed to execute 'hadoop version 2>&1'; the command was
> either not found or exited with a non-zero exit status: 127
> [07:02:46] : [Step 10/10] I1127 07:02:46.425452 2919 local_puller.cpp:94]
> Creating local puller with docker registry '/tmp/R6OUei/registry'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427258 2934
> containerizer.cpp:956] Starting container
> 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 for executor 'test_executor' of
> framework
> [07:02:46] : [Step 10/10] I1127 07:02:46.427592 2938
> metadata_manager.cpp:167] Looking for image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427774 2936 local_puller.cpp:147]
> Untarring image 'test_image_rootfs' from
> '/tmp/R6OUei/registry/test_image_rootfs.tar' to
> '/tmp/R6OUei/store/staging/9krDz2'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512070 2933 local_puller.cpp:167]
> The repositories JSON file for image 'test_image_rootfs' is
> '{"test_image_rootfs":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512279 2933 local_puller.cpp:295]
> Extracting layer tar ball
> '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar
> to rootfs
> '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617442 2937
> metadata_manager.cpp:155] Successfully cached image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617908 2938 provisioner.cpp:286]
> Image layers: 1
> [07:02:46] : [Step 10/10] I1127 07:02:46.617925 2938 provisioner.cpp:296]
> Should hit here
> [07:02:46] : [Step 10/10] I1127 07:02:46.617949 2938 provisioner.cpp:315]
> : bind
> [07:02:46] : [Step 10/10] I1127 07:02:46.617959 2938 provisioner.cpp:315]
> : overlay
> [07:02:46] : [Step 10/10] I1127 07:02:46.617967 2938 provisioner.cpp:315]
> : copy
> [07:02:46] : [Step 10/10] I1127 07:02:46.617974 2938 provisioner.cpp:318]
> Provisioning image rootfs
> '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/rootfses/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7'
> for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 using overlay backend
> [07:02:46] : [Step 10/10] I1127 07:02:46.618408 2936 overlay.cpp:175]
> Created symlink
> '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/links'
> -> '/tmp/DQ3blT'
> [07:02:46] : [Step 10/10] I11
[jira] [Commented] (MESOS-6653) Overlayfs backend may fail to mount the rootfs if both container image and image volume are specified.
[
https://issues.apache.org/jira/browse/MESOS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839028#comment-15839028
]
Gilbert Song commented on MESOS-6653:
-
commit 27a5016154384eb18ec5d7577c1723c9d17e87f7
Author: Gilbert Song
Date: Wed Jan 25 07:51:35 2017 -0800
Fixed overlay backend provisioning multi images symlink.
Since the fix of MESOS-6000, symlinks are used in overlayfs
backend to shorten the arguments when mounting the rootfs.
E.g., '.../backends/overlay/links' is the symlink created
for a provisioned image. It becomes problematic if a
container image is specified while some image volumes are
specified for the same container. An unique symlink is
needed for each image to be provisioned.
Please note that changing the symlinks directory would
still be backward compatible for legacy containers, since
the container backend directory will be removed anyway in
provisioner::destroy().
Review: https://reviews.apache.org/r/54212/
> Overlayfs backend may fail to mount the rootfs if both container image and
> image volume are specified.
> --
>
> Key: MESOS-6653
> URL: https://issues.apache.org/jira/browse/MESOS-6653
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Gilbert Song
>Assignee: Gilbert Song
>Priority: Critical
> Labels: backend, containerizer, overlayfs
>
> Depending on MESOS-6000, we use symlink to shorten the overlayfs mounting
> arguments. However, if more than one image need to be provisioned (e.g., a
> container image is specified while image volumes are specified for the same
> container), the symlink .../backends/overlay/links would fail to be created
> since it exists already.
> Here is a simple log when we hard code overlayfs as our default backend:
> {noformat}
> [07:02:45] : [Step 10/10] [ RUN ]
> Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0
> [07:02:46] : [Step 10/10] I1127 07:02:46.416021 2919
> containerizer.cpp:207] Using isolation:
> filesystem/linux,volume/image,docker/runtime,network/cni
> [07:02:46] : [Step 10/10] I1127 07:02:46.419312 2919
> linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy
> for the Linux launcher
> [07:02:46] : [Step 10/10] E1127 07:02:46.425336 2919 shell.hpp:107]
> Command 'hadoop version 2>&1' failed; this is the output:
> [07:02:46] : [Step 10/10] sh: 1: hadoop: not found
> [07:02:46] : [Step 10/10] I1127 07:02:46.425379 2919 fetcher.cpp:69]
> Skipping URI fetcher plugin 'hadoop' as it could not be created: Failed to
> create HDFS client: Failed to execute 'hadoop version 2>&1'; the command was
> either not found or exited with a non-zero exit status: 127
> [07:02:46] : [Step 10/10] I1127 07:02:46.425452 2919 local_puller.cpp:94]
> Creating local puller with docker registry '/tmp/R6OUei/registry'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427258 2934
> containerizer.cpp:956] Starting container
> 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 for executor 'test_executor' of
> framework
> [07:02:46] : [Step 10/10] I1127 07:02:46.427592 2938
> metadata_manager.cpp:167] Looking for image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427774 2936 local_puller.cpp:147]
> Untarring image 'test_image_rootfs' from
> '/tmp/R6OUei/registry/test_image_rootfs.tar' to
> '/tmp/R6OUei/store/staging/9krDz2'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512070 2933 local_puller.cpp:167]
> The repositories JSON file for image 'test_image_rootfs' is
> '{"test_image_rootfs":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512279 2933 local_puller.cpp:295]
> Extracting layer tar ball
> '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar
> to rootfs
> '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617442 2937
> metadata_manager.cpp:155] Successfully cached image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617908 2938 provisioner.cpp:286]
> Image layers: 1
> [07:02:46] : [Step 10/10] I1127 07:02:46.617925 2938 provisioner.cpp:296]
> Should hit here
> [07:02:46] : [Step 10/10] I1127 07:02:46.617949 2938 provisioner.cpp:315]
> : bind
> [07:02:46] : [Step 10/10] I1127 07:02:46.617959 2938 provisioner.cpp:315]
> : overlay
> [07:02:46] : [Step 10/10] I1127 07:02:46.617967 2938 provisioner.cpp:315]
> : copy
> [07:02:46] : [Step 10/10] I1127 07:02:46.617974 2938 provisioner.cpp:31
[jira] [Commented] (MESOS-6653) Overlayfs backend may fail to mount the rootfs if both container image and image volume are specified.
[
https://issues.apache.org/jira/browse/MESOS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15749117#comment-15749117
]
Jie Yu commented on MESOS-6653:
---
commit 812e5e3d4e4d9e044a1cfe6cc7eaab10efb499b6
Author: Gilbert Song
Date: Wed Dec 14 10:49:19 2016 -0800
Fixed overlay backend symlink error message.
Review: https://reviews.apache.org/r/54211/
> Overlayfs backend may fail to mount the rootfs if both container image and
> image volume are specified.
> --
>
> Key: MESOS-6653
> URL: https://issues.apache.org/jira/browse/MESOS-6653
> Project: Mesos
> Issue Type: Bug
> Components: containerization
>Reporter: Gilbert Song
>Assignee: Gilbert Song
> Labels: backend, containerizer, overlayfs
>
> Depending on MESOS-6000, we use symlink to shorten the overlayfs mounting
> arguments. However, if more than one image need to be provisioned (e.g., a
> container image is specified while image volumes are specified for the same
> container), the symlink .../backends/overlay/links would fail to be created
> since it exists already.
> Here is a simple log when we hard code overlayfs as our default backend:
> {noformat}
> [07:02:45] : [Step 10/10] [ RUN ]
> Nesting/VolumeImageIsolatorTest.ROOT_ImageInVolumeWithRootFilesystem/0
> [07:02:46] : [Step 10/10] I1127 07:02:46.416021 2919
> containerizer.cpp:207] Using isolation:
> filesystem/linux,volume/image,docker/runtime,network/cni
> [07:02:46] : [Step 10/10] I1127 07:02:46.419312 2919
> linux_launcher.cpp:150] Using /sys/fs/cgroup/freezer as the freezer hierarchy
> for the Linux launcher
> [07:02:46] : [Step 10/10] E1127 07:02:46.425336 2919 shell.hpp:107]
> Command 'hadoop version 2>&1' failed; this is the output:
> [07:02:46] : [Step 10/10] sh: 1: hadoop: not found
> [07:02:46] : [Step 10/10] I1127 07:02:46.425379 2919 fetcher.cpp:69]
> Skipping URI fetcher plugin 'hadoop' as it could not be created: Failed to
> create HDFS client: Failed to execute 'hadoop version 2>&1'; the command was
> either not found or exited with a non-zero exit status: 127
> [07:02:46] : [Step 10/10] I1127 07:02:46.425452 2919 local_puller.cpp:94]
> Creating local puller with docker registry '/tmp/R6OUei/registry'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427258 2934
> containerizer.cpp:956] Starting container
> 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 for executor 'test_executor' of
> framework
> [07:02:46] : [Step 10/10] I1127 07:02:46.427592 2938
> metadata_manager.cpp:167] Looking for image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.427774 2936 local_puller.cpp:147]
> Untarring image 'test_image_rootfs' from
> '/tmp/R6OUei/registry/test_image_rootfs.tar' to
> '/tmp/R6OUei/store/staging/9krDz2'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512070 2933 local_puller.cpp:167]
> The repositories JSON file for image 'test_image_rootfs' is
> '{"test_image_rootfs":{"latest":"815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346"}}'
> [07:02:46] : [Step 10/10] I1127 07:02:46.512279 2933 local_puller.cpp:295]
> Extracting layer tar ball
> '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/layer.tar
> to rootfs
> '/tmp/R6OUei/store/staging/9krDz2/815b809d588c80fd6ddf4d6ac244ad1c01ae4cbe0f91cc7480e306671ee9c346/rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617442 2937
> metadata_manager.cpp:155] Successfully cached image 'test_image_rootfs'
> [07:02:46] : [Step 10/10] I1127 07:02:46.617908 2938 provisioner.cpp:286]
> Image layers: 1
> [07:02:46] : [Step 10/10] I1127 07:02:46.617925 2938 provisioner.cpp:296]
> Should hit here
> [07:02:46] : [Step 10/10] I1127 07:02:46.617949 2938 provisioner.cpp:315]
> : bind
> [07:02:46] : [Step 10/10] I1127 07:02:46.617959 2938 provisioner.cpp:315]
> : overlay
> [07:02:46] : [Step 10/10] I1127 07:02:46.617967 2938 provisioner.cpp:315]
> : copy
> [07:02:46] : [Step 10/10] I1127 07:02:46.617974 2938 provisioner.cpp:318]
> Provisioning image rootfs
> '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/rootfses/c71e83d2-5dbe-4eb7-a2fc-b8cc826771f7'
> for container 9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330 using overlay backend
> [07:02:46] : [Step 10/10] I1127 07:02:46.618408 2936 overlay.cpp:175]
> Created symlink
> '/mnt/teamcity/temp/buildTmp/Nesting_VolumeImageIsolatorTest_ROOT_ImageInVolumeWithRootFilesystem_0_1fMo0c/provisioner/containers/9af6c98a-d9f7-4c89-a5ed-fc7ae2fa1330/backends/overlay/links'
> -> '/tmp/DQ3blT'
> [07:02:46] : [Step 10/10] I1127 07:02:4
