I can shepherd no problem. Tim
> On Dec 25, 2015, at 4:32 PM, Shuai Lin <linshuai2...@gmail.com> wrote: > > I'll work on it. @Tim could you shepherd it? > >> On Sat, Dec 26, 2015 at 2:49 AM, Marica Antonacci >> <marica.antona...@ba.infn.it> wrote: >> Hi Tim and Shuai, >> >> thank you very much for your reply. I have opened a JIRA issue on this: >> https://issues.apache.org/jira/browse/MESOS-4249 >> I hope it will be patched soon :) >> >> Best regards, >> Marica >> >> >>> Il giorno 24/dic/2015, alle ore 17:54, Tim Chen <t...@mesosphere.io> ha >>> scritto: >>> >>> Hi Marica/Shuai, >>> >>> Sorry haven't been able to spend the time to repro, but looks like Shuai >>> confirmed it. >>> >>> Can one of you file a JIRA? >>> >>> Thanks! >>> >>> Tim >>> >>>> On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <linshuai2...@gmail.com> wrote: >>>> Hi Marica, >>>> >>>> I can reproduce the problem exactly as you described in the first email of >>>> this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set, >>>> the fetcher works just fine; With it, the fetcher steps seems skipped. >>>> This looks like a bug to me. >>>> >>>> Regards, >>>> Shuai >>>> >>>>> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci >>>>> <marica.antona...@ba.infn.it> wrote: >>>>> Dear all, >>>>> >>>>> I have not solved this issue yet. Please, can anyone run the same test >>>>> and let me know if the fetcher is correctly invoked? >>>>> The test is really simple, just try to start a dockerized app (see json >>>>> definition file below) through marathon on a mesos slave running in a >>>>> docker container started with the option —docker_mesos_image=<mesos slave >>>>> image>. >>>>> I would appreciate very much any feedback. >>>>> >>>>> Sample Marathon app: >>>>> { >>>>> "id": "test-app", >>>>> "container": { >>>>> "type": "DOCKER", >>>>> "docker": { >>>>> "image": "libmesos/ubuntu" >>>>> } >>>>> }, >>>>> "cpus": 1, >>>>> "mem": 512, >>>>> "uris": [ >>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"; >>>>> ], >>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T; >>>>> done" >>>>> } >>>>> >>>>> Docker run command to start dockerized mesos slave: >>>>> >>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e >>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e >>>>> MESOS_CONTAINERIZERS=docker,mesos -e >>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e >>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name >>>>> slave --net host --privileged --pid host mesos-slave >>>>> >>>>> Thank you very much in advance! >>>>> Best regards, >>>>> Marica >>>>> >>>>>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci >>>>>> <marica.antona...@ba.infn.it> ha scritto: >>>>>> >>>>>> Dear Tim, >>>>>> >>>>>> I have collected some information from my test environment, starting the >>>>>> slave container with and without the —docker_mesos_image startup flag. >>>>>> Please let me know if you need further input. Thank you very much for >>>>>> your support! >>>>>> >>>>>> Using the flag —docker_mesos_image: >>>>>> >>>>>> root@mesos-slave:~# docker ps >>>>>> CONTAINER ID IMAGE COMMAND CREATED >>>>>> STATUS PORTS NAMES >>>>>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2 >>>>>> minutes ago Up 2 minutes >>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2 >>>>>> minutes ago Up 2 minutes >>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>>>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3 >>>>>> minutes ago Up 3 minutes slave >>>>>> >>>>>> root@mesos-slave:~# docker logs slave >>>>>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown >>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>>> to user 'root' >>>>>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1; >>>>>> mem(*):32 in work directory >>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d' >>>>>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task >>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor >>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container >>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task >>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor >>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework >>>>>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000' >>>>>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to >>>>>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid' >>>>>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for >>>>>> executor 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from >>>>>> executor(1)@90.147.170.246:39381 >>>>>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container >>>>>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update >>>>>> is identical to existing resources >>>>>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task >>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor >>>>>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at >>>>>> executor(1)@90.147.170.246:39381 >>>>>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update >>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from >>>>>> executor(1)@90.147.170.246:39381 >>>>>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received >>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) >>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing >>>>>> UPDATE for status update TASK_RUNNING (UUID: >>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update >>>>>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050 >>>>>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for >>>>>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) >>>>>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to >>>>>> executor(1)@90.147.170.246:39381 >>>>>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received >>>>>> status update acknowledgement (UUID: >>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing >>>>>> ACK for status update TASK_RUNNING (UUID: >>>>>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> >>>>>> root@mesos-slave:~# docker logs >>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>> --stop_timeout="0ns" >>>>>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos" >>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d" >>>>>> --stop_timeout="0ns" >>>>>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0 >>>>>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave >>>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>>>> Registered docker executor on 90.147.170.246 >>>>>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>> >>>>>> root@mesos-slave:~# docker exec -it >>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor >>>>>> bash >>>>>> root@mesos-slave:/# ls -R /tmp/ >>>>>> hsperfdata_root/ mesos/ >>>>>> root@mesos-slave:/# ls -R /tmp/mesos/ >>>>>> /tmp/mesos/: >>>>>> slaves >>>>>> >>>>>> /tmp/mesos/slaves: >>>>>> db70e09f-f39d-491c-8480-73d9858c140b-S0 >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0: >>>>>> frameworks >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks: >>>>>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000: >>>>>> executors >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors: >>>>>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0: >>>>>> runs >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs: >>>>>> d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>> >>>>>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d: >>>>>> stderr stdout >>>>>> >>>>>> >>>>>> root@mesos-slave:~# docker exec -it >>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>> bash >>>>>> root@mesos-slave:/# env >>>>>> HOSTNAME=mesos-slave >>>>>> HOST=90.147.170.246 >>>>>> PORT0=31220 >>>>>> PORT_10000=31220 >>>>>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 >>>>>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin >>>>>> PWD=/ >>>>>> PORTS=31220 >>>>>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d >>>>>> SHLVL=1 >>>>>> HOME=/ >>>>>> MARATHON_APP_ID=/test-app >>>>>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu >>>>>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z >>>>>> PORT=31220 >>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>> _=/usr/bin/env >>>>>> root@mesos-slave:/# ls -R $MESOS_SANDBOX >>>>>> /mnt/mesos/sandbox: >>>>>> stderr stdout >>>>>> >>>>>> >>>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>>> /var/log/mesos-slave.INFO >>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647 >>>>>> >>>>>> /var/log/mesos: >>>>>> >>>>>> >>>>>> Disabling the flag —docker_mesos_image the fetcher is called, log is >>>>>> created, the file is downloaded in the sandbox: >>>>>> >>>>>> root@mesos-slave:~# docker exec -it slave bash >>>>>> root@mesos-slave:/# >>>>>> root@mesos-slave:/# >>>>>> root@mesos-slave:/# >>>>>> root@mesos-slave:/# ls /var/log/mesos* >>>>>> /var/log/mesos-fetcher.INFO >>>>>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157 >>>>>> /var/log/mesos-slave.INFO >>>>>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124 >>>>>> >>>>>> /var/log/mesos: >>>>>> root@mesos-slave:/# >>>>>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO >>>>>> Log file created at: 2015/12/19 18:16:21 >>>>>> Running on machine: mesos-slave >>>>>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg >>>>>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started! >>>>>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info: >>>>>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/www.stat.cmu.edu\/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"} >>>>>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI >>>>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the >>>>>> sandbox directory >>>>>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI >>>>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from >>>>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>> to >>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv' >>>>>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched >>>>>> 'http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv' >>>>>> to >>>>>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv' >>>>>> root@mesos-slave:/# >>>>>> >>>>>> root@mesos-slave:~# docker exec -it >>>>>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4 >>>>>> bash >>>>>> root@mesos-slave:/# >>>>>> root@mesos-slave:/# >>>>>> root@mesos-slave:/# ls /mnt/mesos/sandbox/ >>>>>> snoqualmie.csv stderr stdout >>>>>> root@mesos-slave:/# >>>>>> >>>>>> >>>>>> Best Regards, >>>>>> Marica >>>>>> >>>>>> >>>>>> >>>>>>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <t...@mesosphere.io> ha >>>>>>> scritto: >>>>>>> >>>>>>> HI Marica, >>>>>>> >>>>>>> Did you see the fetcher invoked at all from the Slave logs? Doesn't >>>>>>> seem possible we don't pass down the URI flags and if the fetcher >>>>>>> failed the container launch should have failed too. >>>>>>> >>>>>>> Also another possible situation is that web UI not really showing the >>>>>>> exact content of the sandbox, can you actually go into the directory >>>>>>> and see if the files are there? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci >>>>>>>> <marica.antona...@ba.infn.it> wrote: >>>>>>>> Hi Tim, >>>>>>>> >>>>>>>> looking at the sandbox I can see only the stderr and stout file (see >>>>>>>> the attached screenshot). If I remove —docker_mesos_image (and >>>>>>>> therefore the executor is run inside the slave container) the file >>>>>>>> specified in the URI field is shown in the sandbox. >>>>>>>> Did you verify that the fetcher is called when using the >>>>>>>> —docker_mesos_image flag ? >>>>>>>> >>>>>>>> Thanks a lot for your feedback >>>>>>>> Marica >>>>>>>> >>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png> >>>>>>>> >>>>>>>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <t...@mesosphere.io> >>>>>>>>> ha scritto: >>>>>>>>> >>>>>>>>> Hi Marica, >>>>>>>>> >>>>>>>>> It should work as we fetch all the files before we launch the >>>>>>>>> executor and place them in the sandbox, and we mount the sandbox into >>>>>>>>> that container as well. >>>>>>>>> >>>>>>>>> How did you verify that the file is not downloaded? >>>>>>>>> >>>>>>>>> Tim >>>>>>>>> >>>>>>>>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci >>>>>>>>>> <marica.antona...@ba.infn.it> wrote: >>>>>>>>>> Hi Grzegorz, >>>>>>>>>> >>>>>>>>>> I’m using this command line for docker run >>>>>>>>>> >>>>>>>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e >>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e >>>>>>>>>> MESOS_CONTAINERIZERS=docker,mesos -e >>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log >>>>>>>>>> -e MESOS_docker_mesos_image=mesos-slave -v >>>>>>>>>> /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos >>>>>>>>>> --name slave --net host --privileged --pid host mesos-slave >>>>>>>>>> >>>>>>>>>> where mesos-slave is the image built from the docker file in this >>>>>>>>>> repo https://github.com/maricaantonacci/mesos-slave-dev >>>>>>>>>> >>>>>>>>>> I have tested successfully the deployment of dockerized applications >>>>>>>>>> through Marathon and dockerized jobs through Chronos and also the >>>>>>>>>> recovery seems to work fine with the flag docker_mesos_image. What >>>>>>>>>> is not working for me is the fetcher: it seems that when the >>>>>>>>>> executor is launched as separate container (thanks to the flag >>>>>>>>>> docker_mesos_image) the information about the URIs to be downloaded >>>>>>>>>> is lost…I hope someone can help to understand if this a bug or I’ m >>>>>>>>>> missing something. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Marica >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk >>>>>>>>>>> <gregor...@gmail.com> ha scritto: >>>>>>>>>>> >>>>>>>>>>> I've tried to use this flag, but cannot really run any container >>>>>>>>>>> when this flag is set. >>>>>>>>>>> I've raised this issue here: >>>>>>>>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html >>>>>>>>>>> and here: >>>>>>>>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351 >>>>>>>>>>> but sadly no one was able to help me... >>>>>>>>>>> >>>>>>>>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci >>>>>>>>>>> <marica.antona...@ba.infn.it> napisał: >>>>>>>>>>>> OK, the problem I spotted is related to the usage of the flag >>>>>>>>>>>> —docker_mesos_image that allows the executor to >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> --docker_mesos_image=VALUE The docker image used to launch this >>>>>>>>>>>> mesos slave instance. If an image is specified, the docker >>>>>>>>>>>> containerizer assumes the slave is running in a docker container, >>>>>>>>>>>> and launches executors with docker containers in order to recover >>>>>>>>>>>> them when the slave restarts and recovers. >>>>>>>>>>>> Has anyone used this flag and tested the behavior of the fetcher? >>>>>>>>>>>> >>>>>>>>>>>> Thank you >>>>>>>>>>>> Marica >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao >>>>>>>>>>>>> <xia...@gmail.com> ha scritto: >>>>>>>>>>>>> >>>>>>>>>>>>> no docker_mesos_image flag in my docker run, and the docker >>>>>>>>>>>>> image is build by myself. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci >>>>>>>>>>>>> <marica.antona...@ba.infn.it>: >>>>>>>>>>>> >>>>>>>>>>>>>> Yes, I did check inside the container and the csv file was not >>>>>>>>>>>>>> downloaded as shown also by the app details (see the screenshot >>>>>>>>>>>>>> below). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Are you running your slave with the --docker_mesos_image flag? >>>>>>>>>>>>>> Can you please provide me the docker run command you are using >>>>>>>>>>>>>> to run your dockerized slave? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thank you very much >>>>>>>>>>>> >>>>>>>>>>>>>> Marica >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> <Schermata 2015-12-18 alle 10.15.58.png> >>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao >>>>>>>>>>>>>>> <xia...@gmail.com> ha scritto: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Marica, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> use your test-app json, i can run it correctly, the csv is >>>>>>>>>>>>>>> truely download by mesos slave. please check mesos-master:5050 >>>>>>>>>>>>>>> to check the task detail download files. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> you describe the app container why not found the csv, because >>>>>>>>>>>>>>> the csv is download in slave container's folder, not in app >>>>>>>>>>>>>>> container. so if you run >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> cd $MESOS_SANDBOX; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> the folder in app container is default value: >>>>>>>>>>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> but in real world, the sandbox is in slave container, not in >>>>>>>>>>>>>>> app container. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci >>>>>>>>>>>>>>> <marica.antona...@ba.infn.it>: >>>>>>>>>>>>>>>> Thank you very much, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I’m using a sample application definition file, just for >>>>>>>>>>>>>>>> testing purpose: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>> "id": "test-app", >>>>>>>>>>>>>>>> "container": { >>>>>>>>>>>>>>>> "type": "DOCKER", >>>>>>>>>>>>>>>> "docker": { >>>>>>>>>>>>>>>> "image": "libmesos/ubuntu" >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>> "cpus": 1, >>>>>>>>>>>>>>>> "mem": 512, >>>>>>>>>>>>>>>> "uris": [ >>>>>>>>>>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv"; >>>>>>>>>>>>>>>> ], >>>>>>>>>>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date >>>>>>>>>>>>>>>> -u +%T; done" >>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here is the docker run command line: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e >>>>>>>>>>>>>>>> MESOS_IP=<SLAVE_IP> -e >>>>>>>>>>>>>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos >>>>>>>>>>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \ >>>>>>>>>>>>>>>> -e >>>>>>>>>>>>>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e >>>>>>>>>>>>>>>> MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave >>>>>>>>>>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v >>>>>>>>>>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net >>>>>>>>>>>>>>>> host --privileged --pid host mesos-slave >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> As already mentioned, if I remove the environment variable >>>>>>>>>>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see >>>>>>>>>>>>>>>> the file snoqualmie.csv inside the sandbox. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you again! I’m looking forward to hearing about your >>>>>>>>>>>>>>>> outcomes. >>>>>>>>>>>>>>>> Best regards, >>>>>>>>>>>>>>>> Marica >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao >>>>>>>>>>>>>>>>> <xia...@gmail.com> ha scritto: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Marica, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Could you please give a sample marathon json, i can test it >>>>>>>>>>>>>>>>> asap. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci >>>>>>>>>>>>>>>>> <marica.antona...@ba.infn.it>: >>>>>>>>>>>>>>>>>> No, using the socket: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao >>>>>>>>>>>>>>>>>>> <xia...@gmail.com> ha scritto: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> docker in docker mode? >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci >>>>>>>>>>>>>>>>>>> <marica.antona...@ba.infn.it>: >>>>>>>>>>>>>>>>>>>> Dear all, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon >>>>>>>>>>>>>>>>>>>> applications and Chronos jobs and I have found that if the >>>>>>>>>>>>>>>>>>>> slave is running inside a docker container (using >>>>>>>>>>>>>>>>>>>> docker_mesos_image startup flag) and you submit the >>>>>>>>>>>>>>>>>>>> deployment of a dockerized application or job the fetcher >>>>>>>>>>>>>>>>>>>> step is not performed. On the other hand, if I request the >>>>>>>>>>>>>>>>>>>> deployment of a non-dockerized application, the URIs are >>>>>>>>>>>>>>>>>>>> correctly fetched. Moreover, if I don’t provide the >>>>>>>>>>>>>>>>>>>> docker_mesos_image flag, the fetcher works fine again for >>>>>>>>>>>>>>>>>>>> both dockerized and non-dockerized applications. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Therefore, it seems that the information about the URIs >>>>>>>>>>>>>>>>>>>> gets lost when the dockerized mesos slave spawns the >>>>>>>>>>>>>>>>>>>> executor docker container that in turn launches the >>>>>>>>>>>>>>>>>>>> application docker container…Has anyone seen this problem >>>>>>>>>>>>>>>>>>>> before? I would like to know if there is a workaround or a >>>>>>>>>>>>>>>>>>>> fixing. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks a lot in advance for you help >>>>>>>>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>>>>>>>> Marica >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>>>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 >>>>>>>>>>>>>>> 下午4.55.30.png><屏幕快照 2015-12-18 下午4.55.38.png> >>>>>>>>>>>>>> >>>>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Deshi Xiao >>>>>>>>>>>>> Twitter: xds2000 >>>>>>>>>>>>> E-mail: xiaods(AT)gmail.com >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>>>> Marica ANTONACCI >>>>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>>>> Via Orabona 4 >>>>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>>>> Skype: marica.antonacci >>>>>>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>>>>>> >>>>>>>>>> ---------------------------------------------------------- >>>>>>>>>> Marica ANTONACCI >>>>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>>>> Via Orabona 4 >>>>>>>>>> 70126 Bari - ITALY >>>>>>>>>> Phone +39 080 5443244 >>>>>>>>>> Skype: marica.antonacci >>>>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>>>> >>>>>>>> ---------------------------------------------------------- >>>>>>>> Marica ANTONACCI >>>>>>>> INFN - National Institute of Nuclear Physics >>>>>>>> Via Orabona 4 >>>>>>>> 70126 Bari - ITALY >>>>>>>> Phone +39 080 5443244 >>>>>>>> Skype: marica.antonacci >>>>>>>> e-mail marica.antona...@ba.infn.it >>>>>> >>>>>> ---------------------------------------------------------- >>>>>> Marica ANTONACCI >>>>>> INFN - National Institute of Nuclear Physics >>>>>> Via Orabona 4 >>>>>> 70126 Bari - ITALY >>>>>> Phone +39 080 5443244 >>>>>> Skype: marica.antonacci >>>>>> e-mail marica.antona...@ba.infn.it >>>>> >>>>> ---------------------------------------------------------- >>>>> Marica ANTONACCI >>>>> INFN - National Institute of Nuclear Physics >>>>> Via Orabona 4 >>>>> 70126 Bari - ITALY >>>>> Phone +39 080 5443244 >>>>> Skype: marica.antonacci >>>>> e-mail marica.antona...@ba.infn.it >> >> ---------------------------------------------------------- >> Marica ANTONACCI >> INFN - National Institute of Nuclear Physics >> Via Orabona 4 >> 70126 Bari - ITALY >> Phone +39 080 5443244 >> Skype: marica.antonacci >> e-mail marica.antona...@ba.infn.it >
