Hi Marica/Shuai,
Sorry haven't been able to spend the time to repro, but looks like Shuai
confirmed it.
Can one of you file a JIRA?
Thanks!
Tim
On Thu, Dec 24, 2015 at 6:16 AM, Shuai Lin <linshuai2...@gmail.com> wrote:
> Hi Marica,
>
> I can reproduce the problem exactly as you described in the first email of
> this thread. Without `MESOS_DOCKER_MESOS_IMAGE` environment variable set,
> the fetcher works just fine; With it, the fetcher steps seems skipped. This
> looks like a bug to me.
>
> Regards,
> Shuai
>
> On Tue, Dec 22, 2015 at 7:41 PM, Marica Antonacci <
> marica.antona...@ba.infn.it> wrote:
>
>> Dear all,
>>
>> I have not solved this issue yet. Please, can anyone run the same test
>> and let me know if the fetcher is correctly invoked?
>> The test is really simple, just try to start a dockerized app (see json
>> definition file below) through marathon on a mesos slave running in a
>> docker container started with the option —docker_mesos_image=<mesos slave
>> image>.
>> I would appreciate very much any feedback.
>>
>> Sample Marathon app:
>> {
>> "id": "test-app",
>> "container": {
>> "type": "DOCKER",
>> "docker": {
>> "image": "libmesos/ubuntu"
>> }
>> },
>> "cpus": 1,
>> "mem": 512,
>> "uris": [ "
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv";
>> ],
>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u +%T;
>> done"
>> }
>>
>> Docker run command to start dockerized mesos slave:
>>
>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos -e
>> MESOS_CONTAINERIZERS=docker,mesos
>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos
>> --name slave --net host --privileged --pid host mesos-slave
>>
>> Thank you very much in advance!
>> Best regards,
>> Marica
>>
>> Il giorno 19/dic/2015, alle ore 19:32, Marica Antonacci <
>> marica.antona...@ba.infn.it> ha scritto:
>>
>> Dear Tim,
>>
>> I have collected some information from my test environment, starting the
>> slave container with and without the —docker_mesos_image startup flag.
>> Please let me know if you need further input. Thank you very much for your
>> support!
>>
>> Using the flag —docker_mesos_image:
>>
>> root@mesos-slave:~# docker ps
>> CONTAINER ID IMAGE COMMAND CREATED
>> STATUS PORTS NAMES
>> b30cea22a07c libmesos/ubuntu "/bin/sh -c 'cd $MESO" 2
>> minutes ago Up 2 minutes
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> da9c78ec5727 mesos-slave "/bin/sh -c '/usr/lib" 2
>> minutes ago Up 2 minutes
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> 150f78fbf327 mesos-slave "/entrypoint.sh /usr/" 3
>> minutes ago Up 3 minutes slave
>>
>> *root@mesos-slave:~# docker logs slave*
>> I1219 18:03:38.308544 19476 slave.cpp:1294] Got assigned task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.314268 19476 slave.cpp:1410] Launching task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 for framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.316261 19476 paths.cpp:436] Trying to chown
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> to user 'root'
>> I1219 18:03:38.327221 19476 slave.cpp:4999] Launching executor
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 with resources cpus(*):0.1;
>> mem(*):32 in work directory
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d'
>> I1219 18:03:38.330817 19476 slave.cpp:1628] Queuing task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' for executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:38.335907 19480 docker.cpp:762] Starting container
>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' for task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' (and executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0') of framework
>> '246b272b-d649-47c0-88ca-6b1ff35f437a-0000'
>> I1219 18:03:40.084027 19482 docker.cpp:386] Checkpointing pid 19590 to
>> '/tmp/mesos/meta/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d/pids/forked.pid'
>> I1219 18:03:40.189172 19479 slave.cpp:2405] Got registration for executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:40.191141 19479 docker.cpp:1012] Ignoring updating container
>> 'd965f59b-cc1a-4081-95d2-f3370214c84d' with resources passed to update is
>> identical to existing resources
>> I1219 18:03:40.192512 19479 slave.cpp:1793] Sending queued task
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' to executor
>> 'test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0' of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 at executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:42.425606 19483 slave.cpp:2762] Handling status update
>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 from executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:42.426415 19483 status_update_manager.cpp:322] Received
>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.427276 19483 status_update_manager.cpp:826] Checkpointing
>> UPDATE for status update TASK_RUNNING (UUID:
>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.521395 19483 slave.cpp:3087] Forwarding the update
>> TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to master@90.147.102.209:5050
>> I1219 18:03:42.521852 19483 slave.cpp:3011] Sending acknowledgement for
>> status update TASK_RUNNING (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for
>> task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000 to executor(1)@
>> 90.147.170.246:39381
>> I1219 18:03:42.576373 19481 status_update_manager.cpp:394] Received
>> status update acknowledgement (UUID: 8348c86f-f89a-4d45-afcb-0f5ea58ea552)
>> for task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>> I1219 18:03:42.577263 19481 status_update_manager.cpp:826] Checkpointing
>> ACK for status update TASK_RUNNING (UUID:
>> 8348c86f-f89a-4d45-afcb-0f5ea58ea552) for task
>> test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0 of framework
>> 246b272b-d649-47c0-88ca-6b1ff35f437a-0000
>>
>> root@mesos-slave:~# *docker logs
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor*
>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>> --logbufsecs="0" --logging_level="INFO"
>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --stop_timeout="0ns"
>> --container="mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false"
>> --initialize_driver_logging="true" --launcher_dir="/usr/libexec/mesos"
>> --logbufsecs="0" --logging_level="INFO"
>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false"
>> --sandbox_directory="/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d"
>> --stop_timeout="0ns"
>> I1219 18:03:40.177598 6 exec.cpp:136] Version: 0.26.0
>> I1219 18:03:40.192060 10 exec.cpp:210] Executor registered on slave
>> db70e09f-f39d-491c-8480-73d9858c140b-S0
>> Registered docker executor on 90.147.170.246
>> Starting task test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>>
>> root@mesos-slave:~# *docker exec -it
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d.executor
>> bash*
>> root@mesos-slave:/# ls -R /tmp/
>> hsperfdata_root/ mesos/
>> root@mesos-slave:/# ls -R /tmp/mesos/
>> /tmp/mesos/:
>> *slaves*
>>
>> /tmp/mesos/slaves:
>> *db70e09f-f39d-491c-8480-73d9858c140b-S0*
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0:
>> *frameworks*
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks:
>> *246b272b-d649-47c0-88ca-6b1ff35f437a-0000*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000:
>> *executors*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors:
>> *test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0:
>> *runs*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs:
>> *d965f59b-cc1a-4081-95d2-f3370214c84d*
>>
>>
>> /tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S0/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0/runs/d965f59b-cc1a-4081-95d2-f3370214c84d:
>> stderr stdout
>>
>>
>> root@mesos-slave:~# *docker exec -it
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> bash*
>> root@mesos-slave:/# env
>> HOSTNAME=mesos-slave
>> HOST=90.147.170.246
>> PORT0=31220
>> PORT_10000=31220
>> MESOS_TASK_ID=test-app.d4398af9-a67a-11e5-b1cf-fa163e920cd0
>> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
>> PWD=/
>> PORTS=31220
>>
>> MESOS_CONTAINER_NAME=mesos-db70e09f-f39d-491c-8480-73d9858c140b-S0.d965f59b-cc1a-4081-95d2-f3370214c84d
>> SHLVL=1
>> HOME=/
>> MARATHON_APP_ID=/test-app
>> MARATHON_APP_DOCKER_IMAGE=libmesos/ubuntu
>> MARATHON_APP_VERSION=2015-12-19T18:03:37.542Z
>> PORT=31220
>> MESOS_SANDBOX=/mnt/mesos/sandbox
>> _=/usr/bin/env
>> root@mesos-slave:/# ls -R $MESOS_SANDBOX
>> /mnt/mesos/sandbox:
>> stderr stdout
>>
>>
>> root@mesos-slave:/# ls /var/log/mesos*
>> */var/log/mesos-slave.INFO*
>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-182512.20647
>>
>> /var/log/mesos:
>>
>>
>> Disabling the flag —docker_mesos_image the fetcher is called, log is
>> created, the file is downloaded in the sandbox:
>>
>> root@mesos-slave:~# docker exec -it slave bash
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/# ls /var/log/mesos*
>> */var/log/mesos-fetcher.INFO*
>> /var/log/mesos-fetcher.mesos-slave.invalid-user.log.INFO.20151219-181621.20157
>> */var/log/mesos-slave.INFO*
>> /var/log/mesos-slave.mesos-slave.invalid-user.log.INFO.20151219-181612.20124
>>
>> /var/log/mesos:
>> root@mesos-slave:/#
>> root@mesos-slave:/# cat /var/log/mesos-fetcher.INFO
>> Log file created at: 2015/12/19 18:16:21
>> Running on machine: mesos-slave
>> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
>> I1219 18:16:21.127075 20157 logging.cpp:172] INFO level logging started!
>> I1219 18:16:21.127499 20157 fetcher.cpp:422] Fetcher Info:
>> {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1","items":[{"action":"BYPASS_CACHE","uri":{"extract":false,"value":"http:\/\/
>> www.stat.cmu.edu
>> \/~cshalizi\/402\/lectures\/16-glm-practicals\/snoqualmie.csv"}}],"sandbox_directory":"\/tmp\/mesos\/slaves\/db70e09f-f39d-491c-8480-73d9858c140b-S1\/frameworks\/246b272b-d649-47c0-88ca-6b1ff35f437a-0000\/executors\/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0\/runs\/1862ced9-3659-4429-8bc6-3f47f58970b4"}
>> I1219 18:16:21.129547 20157 fetcher.cpp:377] Fetching URI '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> I1219 18:16:21.129576 20157 fetcher.cpp:248] Fetching directly into the
>> sandbox directory
>> I1219 18:16:21.129607 20157 fetcher.cpp:185] Fetching URI '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> I1219 18:16:21.129636 20157 fetcher.cpp:132] Downloading resource from '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> to
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>> I1219 18:16:23.221889 20157 fetcher.cpp:454] Fetched '
>> http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv'
>> to
>> '/tmp/mesos/slaves/db70e09f-f39d-491c-8480-73d9858c140b-S1/frameworks/246b272b-d649-47c0-88ca-6b1ff35f437a-0000/executors/test-app.9aee927a-a67c-11e5-b1cf-fa163e920cd0/runs/1862ced9-3659-4429-8bc6-3f47f58970b4/snoqualmie.csv'
>> root@mesos-slave:/#
>>
>> root@mesos-slave:~# docker exec -it
>> mesos-db70e09f-f39d-491c-8480-73d9858c140b-S1.1862ced9-3659-4429-8bc6-3f47f58970b4
>> bash
>> root@mesos-slave:/#
>> root@mesos-slave:/#
>> root@mesos-slave:/# ls /mnt/mesos/sandbox/
>> snoqualmie.csv stderr stdout
>> root@mesos-slave:/#
>>
>>
>> Best Regards,
>> Marica
>>
>>
>>
>> Il giorno 19/dic/2015, alle ore 02:12, Tim Chen <t...@mesosphere.io> ha
>> scritto:
>>
>> HI Marica,
>>
>> Did you see the fetcher invoked at all from the Slave logs? Doesn't seem
>> possible we don't pass down the URI flags and if the fetcher failed the
>> container launch should have failed too.
>>
>> Also another possible situation is that web UI not really showing the
>> exact content of the sandbox, can you actually go into the directory and
>> see if the files are there?
>>
>> Thanks,
>>
>> Tim
>>
>> On Fri, Dec 18, 2015 at 4:23 PM, Marica Antonacci <
>> marica.antona...@ba.infn.it> wrote:
>>
>>> Hi Tim,
>>>
>>> looking at the sandbox I can see only the stderr and stout file (see the
>>> attached screenshot). If I remove —docker_mesos_image (and therefore the
>>> executor is run inside the slave container) the file specified in the URI
>>> field is shown in the sandbox.
>>> Did you verify that the fetcher is called when using the
>>> —docker_mesos_image flag ?
>>>
>>> Thanks a lot for your feedback
>>> Marica
>>>
>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>
>>> Il giorno 19/dic/2015, alle ore 00:25, Tim Chen <t...@mesosphere.io> ha
>>> scritto:
>>>
>>> Hi Marica,
>>>
>>> It should work as we fetch all the files before we launch the executor
>>> and place them in the sandbox, and we mount the sandbox into that container
>>> as well.
>>>
>>> How did you verify that the file is not downloaded?
>>>
>>> Tim
>>>
>>> On Fri, Dec 18, 2015 at 5:26 AM, Marica Antonacci <
>>> marica.antona...@ba.infn.it> wrote:
>>>
>>>> Hi Grzegorz,
>>>>
>>>> I’m using this command line for docker run
>>>>
>>>> # docker run -d MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP> -e
>>>> MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>> -e MESOS_CONTAINERIZERS=docker,mesos -e
>>>> MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins -e MESOS_LOG_DIR=/var/log -e
>>>> MESOS_docker_mesos_image=mesos-slave -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>> /var/run/docker.sock:/var/run/docker.sock -v /tmp/mesos:/tmp/mesos --name
>>>> slave --net host --privileged --pid host mesos-slave
>>>>
>>>> where mesos-slave is the image built from the docker file in this repo
>>>> https://github.com/maricaantonacci/mesos-slave-dev
>>>>
>>>> I have tested successfully the deployment of dockerized applications
>>>> through Marathon and dockerized jobs through Chronos and also the recovery
>>>> seems to work fine with the flag docker_mesos_image. What is not working
>>>> for me is the fetcher: it seems that when the executor is launched as
>>>> separate container (thanks to the flag docker_mesos_image) the information
>>>> about the URIs to be downloaded is lost…I hope someone can help to
>>>> understand if this a bug or I’ m missing something.
>>>>
>>>> Cheers,
>>>> Marica
>>>>
>>>>
>>>>
>>>> Il giorno 18/dic/2015, alle ore 12:11, Grzegorz Graczyk <
>>>> gregor...@gmail.com> ha scritto:
>>>>
>>>> I've tried to use this flag, but cannot really run any container when
>>>> this flag is set.
>>>> I've raised this issue here:
>>>> https://www.mail-archive.com/user@mesos.apache.org/msg04975.html and
>>>> here:
>>>> https://github.com/mesosphere/docker-containers/issues/6#issuecomment-155364351
>>>> but
>>>> sadly no one was able to help me...
>>>>
>>>> pt., 18.12.2015 o 11:33 użytkownik Marica Antonacci <
>>>> marica.antona...@ba.infn.it> napisał:
>>>>
>>>>> OK, the problem I spotted is related to the usage of the
>>>>> flag —docker_mesos_image that allows the executor to
>>>>>
>>>>>
>>>>> --docker_mesos_image=VALUEThe docker image used to launch this mesos
>>>>> slave instance. If an image is specified, the docker containerizer assumes
>>>>> the slave is running in a docker container, and launches executors with
>>>>> docker containers in order to recover them when the slave restarts and
>>>>> recovers.
>>>>> Has anyone used this flag and tested the behavior of the fetcher?
>>>>>
>>>>> Thank you
>>>>> Marica
>>>>>
>>>>>
>>>>> Il giorno 18/dic/2015, alle ore 10:38, tommy xiao <xia...@gmail.com>
>>>>> ha scritto:
>>>>>
>>>>> no docker_mesos_image flag in my docker run, and the docker image is
>>>>> build by myself.
>>>>>
>>>>>
>>>>>
>>>>> 2015-12-18 17:20 GMT+08:00 Marica Antonacci <
>>>>> marica.antona...@ba.infn.it>:
>>>>>
>>>>> Yes, I did check inside the container and the csv file was not
>>>>>> downloaded as shown also by the app details (see the screenshot below).
>>>>>>
>>>>>> Are you running your slave with the --docker_mesos_image flag? Can
>>>>>> you please provide me the docker run command you are using to run your
>>>>>> dockerized slave?
>>>>>>
>>>>>> Thank you very much
>>>>>>
>>>>> Marica
>>>>>>
>>>>>>
>>>>>> <Schermata 2015-12-18 alle 10.15.58.png>
>>>>>>
>>>>>
>>>>>>
>>>>>> Il giorno 18/dic/2015, alle ore 10:00, tommy xiao <xia...@gmail.com>
>>>>>> ha scritto:
>>>>>>
>>>>>> Hi Marica,
>>>>>>
>>>>>> use your test-app json, i can run it correctly, the csv is truely
>>>>>> download by mesos slave. please check mesos-master:5050 to check the task
>>>>>> detail download files.
>>>>>>
>>>>>> you describe the app container why not found the csv, because the csv
>>>>>> is download in slave container's folder, not in app container. so if you
>>>>>> run
>>>>>>
>>>>>> cd $MESOS_SANDBOX;
>>>>>>
>>>>>> the folder in app container is default value:
>>>>>>
>>>>>> MESOS_SANDBOX=/mnt/mesos/sandbox
>>>>>> but in real world, the sandbox is in slave container, not in app
>>>>>> container.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-12-18 16:11 GMT+08:00 Marica Antonacci <
>>>>>> marica.antona...@ba.infn.it>:
>>>>>>
>>>>>>> Thank you very much,
>>>>>>>
>>>>>>> I’m using a sample application definition file, just for testing
>>>>>>> purpose:
>>>>>>>
>>>>>>> {
>>>>>>> "id": "test-app",
>>>>>>> "container": {
>>>>>>> "type": "DOCKER",
>>>>>>> "docker": {
>>>>>>> "image": "libmesos/ubuntu"
>>>>>>> }
>>>>>>> },
>>>>>>> "cpus": 1,
>>>>>>> "mem": 512,
>>>>>>> *"uris": [
>>>>>>> "http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv
>>>>>>> <http://www.stat.cmu.edu/~cshalizi/402/lectures/16-glm-practicals/snoqualmie.csv>"
>>>>>>> ],*
>>>>>>> "cmd": "cd $MESOS_SANDBOX; ls -latr; while sleep 10; do date -u
>>>>>>> +%T; done"
>>>>>>> }
>>>>>>>
>>>>>>> Here is the docker run command line:
>>>>>>>
>>>>>>> # docker run -d -e MESOS_HOSTNAME=<SLAVE_IP> -e MESOS_IP=<SLAVE_IP>
>>>>>>> -e MESOS_MASTER=zk://<node-1>:2181,<node-2>:2181,<node-3>:2181/mesos
>>>>>>> -e MESOS_CONTAINERIZERS=docker,mesos \
>>>>>>> -e MESOS_EXECUTOR_REGISTRATION_TIMEOUT=5mins
>>>>>>> -e MESOS_LOG_DIR=/var/log -e MESOS_docker_mesos_image=mesos-slave
>>>>>>> -v /sys/fs/cgroup:/sys/fs/cgroup -v
>>>>>>> /var/run/docker.sock:/var/run/docker.sock --name slave --net host
>>>>>>> --privileged --pid host mesos-slave
>>>>>>>
>>>>>>>
>>>>>>> As already mentioned, if I remove the environment variable
>>>>>>> MESOS_docker_mesos_image the fetcher works fine and I can see the file
>>>>>>> snoqualmie.csv inside the sandbox.
>>>>>>>
>>>>>>> Thank you again! I’m looking forward to hearing about your outcomes.
>>>>>>> Best regards,
>>>>>>> Marica
>>>>>>>
>>>>>>>
>>>>>>> Il giorno 18/dic/2015, alle ore 04:51, tommy xiao <xia...@gmail.com>
>>>>>>> ha scritto:
>>>>>>>
>>>>>>> Hi Marica,
>>>>>>>
>>>>>>> Could you please give a sample marathon json, i can test it asap.
>>>>>>>
>>>>>>> 2015-12-18 5:11 GMT+08:00 Marica Antonacci <
>>>>>>> marica.antona...@ba.infn.it>:
>>>>>>>
>>>>>>>> No, using the socket:
>>>>>>>>
>>>>>>>> -v /var/run/docker.sock:/var/run/docker.sock
>>>>>>>>
>>>>>>>>
>>>>>>>> Il giorno 17/dic/2015, alle ore 18:07, tommy xiao <xia...@gmail.com>
>>>>>>>> ha scritto:
>>>>>>>>
>>>>>>>> docker in docker mode?
>>>>>>>>
>>>>>>>> 2015-12-17 19:08 GMT+08:00 Marica Antonacci <
>>>>>>>> marica.antona...@ba.infn.it>:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I'm testing the URIs fetching mechanism for both Marathon
>>>>>>>>> applications and Chronos jobs and I have found that if the slave is
>>>>>>>>> running
>>>>>>>>> inside a docker container (using *docker_mesos_image* startup
>>>>>>>>> flag) and you submit the deployment of a dockerized application or
>>>>>>>>> job the
>>>>>>>>> fetcher step is not performed. On the other hand, if I request the
>>>>>>>>> deployment of a non-dockerized application, the URIs are correctly
>>>>>>>>> fetched.
>>>>>>>>> Moreover, if I don’t provide the docker_mesos_image flag, the fetcher
>>>>>>>>> works
>>>>>>>>> fine again for both dockerized and non-dockerized applications.
>>>>>>>>>
>>>>>>>>> Therefore, it seems that the information about the URIs gets lost
>>>>>>>>> when the dockerized mesos slave spawns the executor docker container
>>>>>>>>> that
>>>>>>>>> in turn launches the application docker container…Has anyone seen this
>>>>>>>>> problem before? I would like to know if there is a workaround or a
>>>>>>>>> fixing.
>>>>>>>>>
>>>>>>>>> Thanks a lot in advance for you help
>>>>>>>>> Best Regards,
>>>>>>>>> Marica
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----------------------------------------------------------
>>>>>>>>>
>>>>>>>>> Marica ANTONACCI
>>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>>> Via Orabona 4
>>>>>>>>> 70126 Bari - ITALY
>>>>>>>>> Phone +39 080 5443244
>>>>>>>>> Skype: marica.antonacci
>>>>>>>>> e-mail marica.antona...@ba.infn.it
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Deshi Xiao
>>>>>>>> Twitter: xds2000
>>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>>
>>>>>>>>
>>>>>>>> ----------------------------------------------------------
>>>>>>>>
>>>>>>>> Marica ANTONACCI
>>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>>> Via Orabona 4
>>>>>>>> 70126 Bari - ITALY
>>>>>>>> Phone +39 080 5443244
>>>>>>>> Skype: marica.antonacci
>>>>>>>> e-mail marica.antona...@ba.infn.it
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Deshi Xiao
>>>>>>> Twitter: xds2000
>>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>>>
>>>>>>>
>>>>>>> ----------------------------------------------------------
>>>>>>>
>>>>>>> Marica ANTONACCI
>>>>>>> INFN - National Institute of Nuclear Physics
>>>>>>> Via Orabona 4
>>>>>>> 70126 Bari - ITALY
>>>>>>> Phone +39 080 5443244
>>>>>>> Skype: marica.antonacci
>>>>>>> e-mail marica.antona...@ba.infn.it
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Deshi Xiao
>>>>>> Twitter: xds2000
>>>>>> E-mail: xiaods(AT)gmail.com
>>>>>> <屏幕快照 2015-12-18 下午4.55.16.png><屏幕快照 2015-12-18 下午4.55.30.png><屏幕快照
>>>>>> 2015-12-18 下午4.55.38.png>
>>>>>>
>>>>>>
>>>>>> ----------------------------------------------------------
>>>>>>
>>>>>> Marica ANTONACCI
>>>>>> INFN - National Institute of Nuclear Physics
>>>>>> Via Orabona 4
>>>>>> 70126 Bari - ITALY
>>>>>> Phone +39 080 5443244
>>>>>> Skype: marica.antonacci
>>>>>> e-mail marica.antona...@ba.infn.it
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Deshi Xiao
>>>>> Twitter: xds2000
>>>>> E-mail: xiaods(AT)gmail.com
>>>>>
>>>>>
>>>>> ----------------------------------------------------------
>>>>>
>>>>> Marica ANTONACCI
>>>>> INFN - National Institute of Nuclear Physics
>>>>> Via Orabona 4
>>>>> 70126 Bari - ITALY
>>>>> Phone +39 080 5443244
>>>>> Skype: marica.antonacci
>>>>> e-mail marica.antona...@ba.infn.it
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> ----------------------------------------------------------
>>>>
>>>> Marica ANTONACCI
>>>> INFN - National Institute of Nuclear Physics
>>>> Via Orabona 4
>>>> 70126 Bari - ITALY
>>>> Phone +39 080 5443244
>>>> Skype: marica.antonacci
>>>> e-mail marica.antona...@ba.infn.it
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>> ----------------------------------------------------------
>>>
>>> Marica ANTONACCI
>>> INFN - National Institute of Nuclear Physics
>>> Via Orabona 4
>>> 70126 Bari - ITALY
>>> Phone +39 080 5443244
>>> Skype: marica.antonacci
>>> e-mail marica.antona...@ba.infn.it
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antona...@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> ----------------------------------------------------------
>>
>> Marica ANTONACCI
>> INFN - National Institute of Nuclear Physics
>> Via Orabona 4
>> 70126 Bari - ITALY
>> Phone +39 080 5443244
>> Skype: marica.antonacci
>> e-mail marica.antona...@ba.infn.it
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
