Olivier,
If it is not urgent, I think that you can use 0.29 in next week, the
unified container will have more features in 0.29 such as external storage
and network integration.
Thanks,
Guangya
On Tue, May 24, 2016 at 2:17 AM, Olivier Sallou <olivier.sal...@irisa.fr>
wrote:
>
>
> ----- Mail original -----
> > De: "Guangya Liu" <gyliu...@gmail.com>
> > À: "dev" <dev@mesos.apache.org>, "Jie Yu" <j...@mesosphere.io>, "Gilbert
> Song" <gilb...@mesosphere.io>
> > Envoyé: Lundi 23 Mai 2016 17:34:41
> > Objet: Re: volume / mount point error with Unified Containerizer
> >
> > It is a bit strange to me, I also did some test and review code for
> > relative path, and found that relative path works well.
> >
> > In 0.28.1, if deploy a docker container with MesosContaineirizer, then if
> > using absolute path as continer_path, the mesos agent will update the
> > container_path to a relative path by adding a prefix ./rootfs to the
> > container_path, e.g. /file/path = > ./rootfs/file/path.
> >
> > If deploy a docker container with MesosContaineirizer with relative path
> as
> > container_path, then the mesos agent will not update the container_path.
> >
> > So the final mount point for the container should be either
> >
> > 1) /tmp/mesos/slaves/agent_id/frameworks/framework_id/
> > executors/51/runs/container_id/.rootfs/file/path
> > 2) /tmp/mesos/slaves/agent_id/frameworks/framework_id/executors/51/runs/
> > container_id/file/path
> >
> > The only difference is adding ./rootfs as a prefix or not, the test
> result
> > is that 1) does not work and 2) works well. And even the mount for 1)
> > failed, but I can see the mount point path does exist.
> >
>
> @Guangya
> I confirm that using relative path works fine, I get volumes in mesos path
> (but it does not help for my implementation).
> If I use the Docker containerizer, absolute paths are fine, this is what I
> use for the moment in my code, and am investigating to switch to unified
> container.
>
>
> > @Yu Jie and @Gilbert, any comments for this?
> >
> > @Oilivier,
> >
> > In order not to block your test, can you please use mesos after 0.28.1?
> You
> > can use either 0.28.2 or above version.
>
> Well, as this is not an urgent matter, I am waiting 0.29 to test against
> this release (with other features I am waiting for).
>
> >
> > Thanks,
> >
> > Guangya
> >
> >
> > On Mon, May 23, 2016 at 10:30 PM, Guangya Liu <gyliu...@gmail.com>
> wrote:
> >
> > > Thanks Olivier, I can reproduce this issue now and still checking what
> is
> > > wrong.
> > >
> > > What I did is as following:
> > > 1) Check out code with tag of 0.28.1
> > > 2) update mesos-execute to add a host path volume
> > > diff --git a/src/cli/execute.cpp b/src/cli/execute.cpp
> > > index 81a0388..0ff913c 100644
> > > --- a/src/cli/execute.cpp
> > > +++ b/src/cli/execute.cpp
> > > @@ -72,6 +72,8 @@ using mesos::v1::TaskID;
> > > using mesos::v1::TaskInfo;
> > > using mesos::v1::TaskState;
> > > using mesos::v1::TaskStatus;
> > > +using mesos::v1::Volume;
> > > +using mesos::v1::Parameters;
> > >
> > > using mesos::v1::scheduler::Call;
> > > using mesos::v1::scheduler::Event;
> > > @@ -572,6 +574,12 @@ private:
> > > }
> > > }
> > >
> > > + Volume* volume1 = containerInfo.add_volumes();
> > > + volume1->set_container_path("/tmp/abcd");
> > > + volume1->set_mode(Volume::RW);
> > > + volume1->set_host_path("/root/convoy");
> > > + cout << "Add Voume 1" << endl;
> > > +
> > > return containerInfo;
> > > } else if (containerizer == "docker") {
> > > // 'docker' containerizer only supports 'docker' images.
> > > 3) launch a task with docker image, task failed.
> > >
> > > 4) Check sandbox:
> > > + /root/src/mesos/m1/mesos/build/src/mesos-containerizer mount
> > > --help=false --operation=make-rslave --path=/
> > > + grep -E /tmp/mesos/.+ /proc/self/mountinfo
> > > + grep -v 3239aafc-78d8-4f70-81e5-f32fb3733339
> > > + cut+ -d -f5
> > > xargs --no-run-if-empty umount -l
> > > + mount -n --rbind
> > >
> /tmp/mesos/provisioner/containers/3239aafc-78d8-4f70-81e5-f32fb3733339/backends/copy/rootfses/5e8bf3fa-53b1-4bd5-bb3d-525ddc7900b6
> > >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs
> > > + mount -n --rbind /root/convoy
> > >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > mount: mount point
> > >
> /tmp/mesos/slaves/a4294ed5-10e8-47db-a3b9-a43a4f951374-S0/frameworks/a4294ed5-10e8-47db-a3b9-a43a4f951374-0000/executors/test/runs/3239aafc-78d8-4f70-81e5-f32fb3733339/.rootfs/tmp/abcd
> > > does not exist
> > > Failed to execute a preparation shell command
> > >
> > > Will check more for this.
> > >
> > > Thanks,
> > >
> > > Guangya
> > >
> > > On Mon, May 23, 2016 at 3:35 PM, Olivier Sallou <
> olivier.sal...@irisa.fr>
> > > wrote:
> > >
> > >>
> > >>
> > >> On 05/23/2016 09:33 AM, Olivier Sallou wrote:
> > >> >
> > >> > On 05/20/2016 03:26 PM, Guangya Liu wrote:
> > >> >> Since you are using docker image which means that your container
> will
> > >> have
> > >> >> rootfs, so it is not required to have the absolute path exist, the
> > >> linux
> > >> >> file system isolator will help create the path automatically
> > >> >>
> > >>
> https://github.com/apache/mesos/blob/0.28.x/src/slave/containerizer/mesos/isolators/filesystem/linux.cpp#L390-L402
> > >> >>
> > >> >> Can you please share your framework? How did you set the volume
> part in
> > >> >> your framework?
> > >> > @Guangya
> > >> >
> > >> > I use Python API.
> > >> >
> > >> > Here is related code:
> > >> >
> > >> > ....
> > >> > # Define container volumes
> > >> > for v in job['container']['volumes']:
> > >> > volume = container.volumes.add()
> > >> > volume.container_path = v['mount']
> > >> > volume.host_path = v['path']
> > >> > if v['acl'] == 'rw':
> > >> > volume.mode = 1 # mesos_pb2.Volume.Mode.RW
> > >> > else:
> > >> > volume.mode = 2 # mesos_pb2.Volume.Mode.RO
> > >> >
> > >> > => In my test case, I add 2 volumes from a host shared
> directory,
> > >> > mounted in container as /mnt/go-docker and /mnt/god-data.
> > >> >
> > >> > ...
> > >> > # Define docker image and network
> > >> > docker = mesos_pb2.ContainerInfo.MesosInfo()
> > >> > docker.image.type = 2 # Docker
> > >> > docker.image.docker.name ='centos:latest'
> > >> > # Request an IP from a network module
> > >> > network_info = container.network_infos.add()
> > >> > network_info_name = 'sampletest'
> > >> > # Get an IP V4 address
> > >> > ip_address = network_info.ip_addresses.add()
> > >> > ip_address.protocol = 1
> > >> > # The network group to join
> > >> > group = network_info.groups.append(network_info_name)
> > >> > port_list = [22]
> > >> > if port_list:
> > >> > for port in port_list:
> > >> > job['container']['port_mapping'].append({'host':
> > >> > port, 'container': port})
> > >> > container.mesos.MergeFrom(docker)
> > >> >
> > >> > It results in error message:
> > >> >
> > >> > + mount -n --rbind
> > >> >
> > >>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > >> >
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > >> > + mount -n --rbind
> > >> >
> > >>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >> >
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> > mount: mount point
> > >> >
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> > does not exist
> > >> > Failed to execute a preparation shell command
> > >> >
> > >> >
> > >> > We can see the .rootfs, but mnt/god-data under .rootfs fails. Local
> > >> > directory exists, it does not pre-exists in container. What is
> strange
> > >> > is , if I look in mesos task dir, .rootfs/mnt/go-data, directory is
> > >> present.
> > >> >
> > >> > Or, is the error message (.rootfs/mnt/god-data does not exist)
> simply a
> > >> > warning, and it creates it, and final error (Failed to execute a
> > >> > preparation shell command) not related (and unclear...)
> > >> Additional info: command to execute in container is located in one of
> > >> mounted volume.
> > >> >
> > >> > Olivier
> > >> >> Thanks,
> > >> >>
> > >> >> Guangya
> > >> >>
> > >> >> On Fri, May 20, 2016 at 4:54 AM, Olivier Sallou <
> > >> olivier.sal...@irisa.fr>
> > >> >> wrote:
> > >> >>
> > >> >>> ----- Mail original -----
> > >> >>>> De: "Gilbert Song" <gilb...@mesosphere.io>
> > >> >>>> À: "dev" <dev@mesos.apache.org>
> > >> >>>> Envoyé: Jeudi 19 Mai 2016 01:57:16
> > >> >>>> Objet: Re: volume / mount point error with Unified Containerizer
> > >> >>>>
> > >> >>>> @Olivier,
> > >> >>>> In mesos 0.28.1, you are supposed to be able bind mount a volume
> from
> > >> >>>> the host into the mesos container. Did you specify a docker
> image (we
> > >> >>>> determine
> > >> >>>> the mount point differently depending whether the container has a
> > >> >>> rootfs)?
> > >> >>>
> > >> >>> Yes I specified an image, a Docker image URI.
> > >> >>>
> > >> >>>> How
> > >> >>>> do you specify your 'container_path' (the mount point in the
> > >> container)?
> > >> >>> If
> > >> >>>> it is an
> > >> >>>> absolute path, we require that dir to be pre-existed. If it is a
> > >> relative
> > >> >>>> path, we will
> > >> >>>> mkdir for it.
> > >> >>> It is an absolute path, but it does not exists in image (this is
> the
> > >> >>> issue). Images are custom Docker images (images containing tools
> for
> > >> batch
> > >> >>> computing), and I want, for example, to mount some shared
> resources
> > >> (user
> > >> >>> home dir, common data, etc.) in the image. Of course those
> > >> directories do
> > >> >>> not pre-exists in container images as they are specific to the
> > >> environment.
> > >> >>> Requiring existence of the directory in the image is not issue as
> it
> > >> >>> prevents using any existing image from a repo.
> > >> >>>
> > >> >>> When using Docker containerizer it works fine, I can mount any
> > >> external
> > >> >>> storage in the container.
> > >> >>>
> > >> >>> Olivie
> > >> >>>
> > >> >>>
> > >> >>>> @Joshua,
> > >> >>>> Thank for posting your workaround on mesos. As I mentioned
> above, in
> > >> >>> 0.28.1
> > >> >>>> or
> > >> >>>> older, we only mkdir for container_path which is relative path
> (not
> > >> >>>> starting with "/").
> > >> >>>> Because if no rootfs specified for a mesos container, the
> container
> > >> >>> shares
> > >> >>>> the host
> > >> >>>> root filesystem. Obviously we don't want any random files to be
> > >> created
> > >> >>>> implicitly
> > >> >>>> on your host fs.
> > >> >>>> From mesos 0.29 (release by the end of this month), we will
> mkdir the
> > >> >>> mount
> > >> >>>> point in the container except for the command task case that
> specify
> > >> an
> > >> >>>> absolute
> > >> >>>> container_path without a rootfs. Because we simplify the mounting
> > >> logic,
> > >> >>> and
> > >> >>>> sandbox bind mount will only be done in container mount namespace
> > >> >>> instead of
> > >> >>>> host mount namespace (what we did before). Please keep tuned.
> > >> >>>>
> > >> >>>> Cheers,
> > >> >>>> Gilbert
> > >> >>>>
> > >> >>>> On Wed, May 18, 2016 at 8:14 AM, Joshua Cohen <jco...@apache.org
> >
> > >> wrote:
> > >> >>>>
> > >> >>>>> Hi Olivier,
> > >> >>>>>
> > >> >>>>> I touched on this issue as part of
> > >> >>>>> https://issues.apache.org/jira/browse/MESOS-5229. It would be
> nice
> > >> if
> > >> >>>>> Mesos
> > >> >>>>> automatically created container mount points if they don't
> already
> > >> >>> exist.
> > >> >>>>> In the meantime, as a workaround for this, I've updated my
> > >> filesystem
> > >> >>>>> images to include the path (e.g. in Dockerfile, add `RUN mkdir
> -p
> > >> >>>>> /some/mount/point`). Not the best solution, but the only thing
> I've
> > >> >>> seen
> > >> >>>>> that works at the moment.
> > >> >>>>>
> > >> >>>>> Cheers,
> > >> >>>>>
> > >> >>>>> Joshua
> > >> >>>>>
> > >> >>>>> On Wed, May 18, 2016 at 7:36 AM, Guangya Liu <
> gyliu...@gmail.com>
> > >> >>> wrote:
> > >> >>>>>> It's pretty simple for you from scratch with source code
> > >> >>>>>>
> > >> >>>>>>
> > >> >>>
> > >>
> https://github.com/apache/mesos/blob/master/docs/getting-started.md#building-mesos
> > >> >>>>>> ;-)
> > >> >>>>>>
> > >> >>>>>> Thanks,
> > >> >>>>>>
> > >> >>>>>> Guangya
> > >> >>>>>>
> > >> >>>>>> On Wed, May 18, 2016 at 8:30 PM, Olivier Sallou <
> > >> >>> olivier.sal...@irisa.fr
> > >> >>>>>> wrote:
> > >> >>>>>>
> > >> >>>>>>> On 05/18/2016 02:31 PM, Guangya Liu wrote:
> > >> >>>>>>>> Just saw that you are working with 0.28.1, the "docker volume
> > >> >>> driver"
> > >> >>>>>>> code
> > >> >>>>>>>> was not in 0.28.1, can you please have a try with mesos
> master
> > >> >>> branch
> > >> >>>>>> if
> > >> >>>>>>>> you are only doing some test?
> > >> >>>>>>> this is indeed test only for the moment. But I will have to
> > >> >>>>>>> recompile/install mesos :-( (I used packages for install).
> > >> >>>>>>>
> > >> >>>>>>> I will try when possible, but thanks for the hint.
> > >> >>>>>>>> Thanks,
> > >> >>>>>>>>
> > >> >>>>>>>> Guangya
> > >> >>>>>>>>
> > >> >>>>>>>> On Wed, May 18, 2016 at 8:28 PM, Guangya Liu <
> gyliu...@gmail.com
> > >> >>>>>> wrote:
> > >> >>>>>>>>> Hi Olivier,
> > >> >>>>>>>>>
> > >> >>>>>>>>> I think that you need to enable "docker volume isolator" if
> you
> > >> >>> want
> > >> >>>>>> use
> > >> >>>>>>>>> external storage with unified container I was writing a
> document
> > >> >>>>> here
> > >> >>>>>>>>> https://reviews.apache.org/r/47511/, perhaps you can have
> a try
> > >> >>>>>>> according
> > >> >>>>>>>>> to the document and post some comments there if you find any
> > >> >>> issues.
> > >> >>>>>>>>> Also you can patch mesos-execute here
> > >> >>>>>>> https://reviews.apache.org/r/46762/ to
> > >> >>>>>>>>> have a try with mesos-execute.
> > >> >>>>>>>>>
> > >> >>>>>>>>> Thanks,
> > >> >>>>>>>>>
> > >> >>>>>>>>> Guangya
> > >> >>>>>>>>>
> > >> >>>>>>>>> On Wed, May 18, 2016 at 7:17 PM, Olivier Sallou <
> > >> >>>>>>> olivier.sal...@irisa.fr>
> > >> >>>>>>>>> wrote:
> > >> >>>>>>>>>
> > >> >>>>>>>>>> Answering (partially) to myself.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> I seems issue is container_path does not exists inside
> > >> >>> container.
> > >> >>>>> On
> > >> >>>>>>>>>> Docker, path is created and mounted. With pure mesos,
> > >> >>>>> container_path
> > >> >>>>>>>>>> must exists.
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> mesos.proto says: "If the path is an absolute path, that
> path
> > >> >>> must
> > >> >>>>>>>>>> already exist."
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> This is an issue however, using Docker images, the path I
> want
> > >> >>> to
> > >> >>>>>> mount
> > >> >>>>>>>>>> does not exists, and it cannot be modified "on the fly".
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> Is there a workaround for this ?
> > >> >>>>>>>>>>
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> On 05/18/2016 12:24 PM, Olivier Sallou wrote:
> > >> >>>>>>>>>>> Hi,
> > >> >>>>>>>>>>> I am trying unified containerizer on a single server
> > >> >>>>> (master/slave)
> > >> >>>>>> on
> > >> >>>>>>>>>>> mesos 0.28.1, to switch from docker containerizer to
> > >> >>> mesos+docker
> > >> >>>>>>> image
> > >> >>>>>>>>>>> container.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I have setup slave config as suggested in documentation:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> containerizers=docker,mesos
> > >> >>>>>>>>>>> image_providers=docker \
> > >> >>>>>>>>>>> isolation=filesystem/linux,docker/runtime
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> However, when I execute my task with a volume I have an
> error:
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> ....
> > >> >>>>>>>>>>> + mount -n --rbind
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /tmp/mesos/provisioner/containers/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/backends/copy/rootfses/f9f66bb2-308d-4555-ba77-49ec61cbeb4f
> > >> >>>
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs
> > >> >>>>>>>>>>> + mount -n --rbind
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >> >>>
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> >>>>>>>>>>> mount: mount point
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /tmp/mesos/slaves/2a296daf-7419-4659-ade1-763c792cd522-S0/frameworks/aef1b0e3-ea2d-4770-baac-96d673ab88f9-0000/executors/51/runs/2d7ea311-5e8b-440f-a3ca-a40e1b946b8e/.rootfs/mnt/god-data
> > >> >>>>>>>>>>> does not exist
> > >> >>>>>>>>>>> Failed to execute a preparation shell command
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Then, my task switches to FAILED.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> I define a local volume to bind mount in my "container"
> > >> >>>>>>>>>>>
> > >> >>>
> > >>
> /home/osallou/Development/NOSAVE/go-docker/godshared/tasks/pairtree_root/us/er/_o/sa/ll/ou/task
> > >> >>>>>>>>>>> => /mnt/god-data
> > >> >>>>>>>>>>> My directory exists on local server.
> > >> >>>>>>>>>>> In mesos UI, I can see the .rootfs directory along stdout
> and
> > >> >>>>> stderr
> > >> >>>>>>>>>>> files, and inside .rootfs, I can see /mnt/god-data
> (empty).
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Running the same using Docker containerizer instead of
> mesos
> > >> >>>>>>>>>>> containerizer (with a Docker image) works fine.
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> It seems it fails to mount my local directory in the
> > >> >>> container.
> > >> >>>>> Any
> > >> >>>>>>> idea
> > >> >>>>>>>>>>> of what is going wrong or how to debug this?
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>>> Thanks
> > >> >>>>>>>>>>>
> > >> >>>>>>>>>> --
> > >> >>>>>>>>>> Olivier Sallou
> > >> >>>>>>>>>> IRISA / University of Rennes 1
> > >> >>>>>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >> >>>>>>>>>> Tel: 02.99.84.71.95
> > >> >>>>>>>>>>
> > >> >>>>>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >> >>>>>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB
> 326D
> > >> >>>>> 8438
> > >> >>>>>>> --
> > >> >>>>>>> Olivier Sallou
> > >> >>>>>>> IRISA / University of Rennes 1
> > >> >>>>>>> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >> >>>>>>> Tel: 02.99.84.71.95
> > >> >>>>>>>
> > >> >>>>>>> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >> >>>>>>> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB
> 326D
> > >> >>> 8438
> > >>
> > >> --
> > >> Olivier Sallou
> > >> IRISA / University of Rennes 1
> > >> Campus de Beaulieu, 35000 RENNES - FRANCE
> > >> Tel: 02.99.84.71.95
> > >>
> > >> gpg key id: 4096R/326D8438 (keyring.debian.org)
> > >> Key fingerprint = 5FB4 6F83 D3B9 5204 6335 D26D 78DC 68DB 326D 8438
> > >>
> > >>
> > >
> >
>
