Yes. The related code is located in https://github.com/apache/mesos/blob/master/src/slave/main.cpp#L123
In fact, environment variables starts with MESOS_ would load as flags variables. https://github.com/apache/mesos/blob/master/3rdparty/libprocess/3rdparty/stout/include/stout/flags/flags.hpp#L52 On Fri, Oct 9, 2015 at 11:33 AM, Jay Taylor <outtat...@gmail.com> wrote: > One question for you haosdent- > > You mentioned that the flags.launcher_dir should propagate to the docker > executor all the way up the chain. Can you show me where this logic is in > the codebase? I didn't see where that was happening and would like to > understand the mechanism. > > Thanks! > Jay > > > > On Oct 8, 2015, at 8:29 PM, Jay Taylor <outtat...@gmail.com> wrote: > > Maybe tomorrow I will build a fresh cluster from scratch to see if the > broken behavior experienced today still persists. > > On Oct 8, 2015, at 7:52 PM, haosdent <haosd...@gmail.com> wrote: > > As far as I know, MESOS_LAUNCHER_DIR is works by set flags.launcher_dir > which would find mesos-docker-executor and mesos-health-check under this > dir. Although the env is not propagated, but MESOS_LAUNCHER_DIR still > works because flags.launcher_dir is get from it. > > For example, because I > ``` > export MESOS_LAUNCHER_DIR=/tmp > ``` > before start mesos-slave. So when I launch slave, I could find this log in > slave log > ``` > I1009 10:27:26.594599 1416 slave.cpp:203] Flags at startup: > xxxxx --launcher_dir="/tmp" > ``` > > And from your log, I not sure why your MESOS_LAUNCHER_DIR become sandbox > dir. Is it because MESOS_LAUNCHER_DIR is overrided in your other scripts? > > > On Fri, Oct 9, 2015 at 1:56 AM, Jay Taylor <outtat...@gmail.com> wrote: > >> I haven't ever changed MESOS_LAUNCHER_DIR/--launcher_dir before. >> >> I just tried setting both the env var and flag on the slaves, and have >> determined that the env var is not present when it is being checked >> src/docker/executor.cpp @ line 573: >> >> const Option<string> envPath = os::getenv("MESOS_LAUNCHER_DIR"); >>> string path = >>> envPath.isSome() ? envPath.get() >>> : os::realpath(Path(argv[0]).dirname()).get(); >>> cout << "MESOS_LAUNCHER_DIR: envpath.isSome()->" << (envPath.isSome() >>> ? "yes" : "no") << endl; >>> cout << "MESOS_LAUNCHER_DIR: path='" << path << "'" << endl; >> >> >> Exported MESOS_LAUNCHER_DIR env var (and verified it is correctly >> propagated along up to the point of mesos-slave launch): >> >> $ cat /etc/default/mesos-slave >>> export >>> MESOS_MASTER="zk://mesos-primary1a:2181,mesos-primary2a:2181,mesos-primary3a:2181/mesos" >>> export MESOS_CONTAINERIZERS="mesos,docker" >>> export MESOS_EXECUTOR_REGISTRATION_TIMEOUT="5mins" >>> export MESOS_PORT="5050" >>> export MESOS_LAUNCHER_DIR="/usr/libexec/mesos" >> >> >> TASK OUTPUT: >> >> >>> *MESOS_LAUNCHER_DIR: envpath.isSome()->no**MESOS_LAUNCHER_DIR: >>> path='/tmp/mesos/slaves/61373c0e-7349-4173-ab8d-9d7b260e8a30-S1/frameworks/20150924-210922-1608624320-5050-1792-0020/executors/hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253/runs/41f8eed6-ec6c-4e6f-b1aa-0a2817a600ad'* >>> Registered docker executor on mesos-worker2a >>> Starting task hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253 >>> Launching health check process: >>> /tmp/mesos/slaves/61373c0e-7349-4173-ab8d-9d7b260e8a30-S1/frameworks/20150924-210922-1608624320-5050-1792-0020/executors/hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253/runs/41f8eed6-ec6c-4e6f-b1aa-0a2817a600ad/mesos-health-check >>> --executor=(1)@192.168.225.59:44523 >>> --health_check_json={"command":{"shell":true,"value":"docker exec >>> mesos-61373c0e-7349-4173-ab8d-9d7b260e8a30-S1.41f8eed6-ec6c-4e6f-b1aa-0a2817a600ad >>> sh -c \" \/bin\/bash >>> \""},"consecutive_failures":3,"delay_seconds":5.0,"grace_period_seconds":10.0,"interval_seconds":10.0,"timeout_seconds":10.0} >>> --task_id=hello-app_web-v3.22f9c7e4-2109-48a9-998e-e116141ec253 >>> Health check process launched at pid: 2519 >> >> >> The env var is not propagated when the docker executor is launched >> in src/slave/containerizer/docker.cpp around line 903: >> >> vector<string> argv; >>> argv.push_back("mesos-docker-executor"); >>> // Construct the mesos-docker-executor using the "name" we gave the >>> // container (to distinguish it from Docker containers not created >>> // by Mesos). >>> Try<Subprocess> s = subprocess( >>> path::join(flags.launcher_dir, "mesos-docker-executor"), >>> argv, >>> Subprocess::PIPE(), >>> Subprocess::PATH(path::join(container->directory, "stdout")), >>> Subprocess::PATH(path::join(container->directory, "stderr")), >>> dockerFlags(flags, container->name(), container->directory), >>> environment, >>> lambda::bind(&setup, container->directory)); >> >> >> A little ways above we can see the environment is setup w/ the container >> tasks defined env vars. >> >> See src/slave/containerizer/docker.cpp around line 871: >> >> // Include any enviroment variables from ExecutorInfo. >>> foreach (const Environment::Variable& variable, >>> container->executor.command().environment().variables()) { >>> environment[variable.name()] = variable.value(); >>> } >> >> >> Should I file a JIRA for this? Have I overlooked anything? >> >> >> On Wed, Oct 7, 2015 at 8:11 PM, haosdent <haosd...@gmail.com> wrote: >> >>> >Not sure what was going on with health-checks in 0.24.0. >>> 0.24.1 should be works. >>> >>> >Do any of you know which host the path >>> "/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check" >>> should exist on? It definitely doesn't exist on the slave, hence execution >>> failing. >>> >>> Does you set MESOS_LAUNCHER_DIR/--launcher_dir incorrectly before? We >>> got mesos-health-check from MESOS_LAUNCHER_DIR/--launcher_id or use the >>> same dir of mesos-docker-executor. >>> >>> On Thu, Oct 8, 2015 at 10:46 AM, Jay Taylor <outtat...@gmail.com> wrote: >>> >>>> Maybe I spoke too soon. >>>> >>>> Now the checks are attempting to run, however the STDERR is not looking >>>> good. I've added some debugging to the error message output to show the >>>> path, argv, and envp variables: >>>> >>>> STDOUT: >>>> >>>> --container="mesos-16b49e90-6852-4c91-8e70-d89c54f25668-S1.73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>> --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO" >>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>> --sandbox_directory="/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>> --stop_timeout="0ns" >>>>> --container="mesos-16b49e90-6852-4c91-8e70-d89c54f25668-S1.73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>> --docker="docker" --docker_socket="/var/run/docker.sock" --help="false" >>>>> --initialize_driver_logging="true" --logbufsecs="0" --logging_level="INFO" >>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>> --sandbox_directory="/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc" >>>>> --stop_timeout="0ns" >>>>> Registered docker executor on mesos-worker2a >>>>> Starting task >>>>> app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0 >>>>> Launching health check process: >>>>> /tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check >>>>> --executor=(1)@192.168.225.59:43917 >>>>> --health_check_json={"command":{"shell":true,"value":"docker exec >>>>> mesos-16b49e90-6852-4c91-8e70-d89c54f25668-S1.73dbfe88-1dbb-4f61-9a52-c365558cdbfc >>>>> sh -c \" exit 1 >>>>> \""},"consecutive_failures":3,"delay_seconds":0.0,"grace_period_seconds":10.0,"interval_seconds":10.0,"timeout_seconds":10.0} >>>>> --task_id=app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0 >>>>> Health check process launched at pid: 3012 >>>> >>>> >>>> STDERR: >>>> >>>> I1008 02:17:28.870434 2770 exec.cpp:134] Version: 0.26.0 >>>>> I1008 02:17:28.871860 2778 exec.cpp:208] Executor registered on slave >>>>> 16b49e90-6852-4c91-8e70-d89c54f25668-S1 >>>>> WARNING: Your kernel does not support swap limit capabilities, memory >>>>> limited without swap. >>>>> ABORT: (src/subprocess.cpp:180): Failed to os::execvpe in childMain >>>>> (path.c_str()='/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check', >>>>> argv='/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check', >>>>> envp=''): No such file or directory*** Aborted at 1444270649 (unix time) >>>>> try "date -d @1444270649" if you are using GNU date *** >>>>> PC: @ 0x7f4a37ec6cc9 (unknown) >>>>> *** SIGABRT (@0xbc4) received by PID 3012 (TID 0x7f4a2f9f6700) from >>>>> PID 3012; stack trace: *** >>>>> @ 0x7f4a38265340 (unknown) >>>>> @ 0x7f4a37ec6cc9 (unknown) >>>>> @ 0x7f4a37eca0d8 (unknown) >>>>> @ 0x4191e2 _Abort() >>>>> @ 0x41921c _Abort() >>>>> @ 0x7f4a39dc2768 process::childMain() >>>>> @ 0x7f4a39dc4f59 std::_Function_handler<>::_M_invoke() >>>>> @ 0x7f4a39dc24fc process::defaultClone() >>>>> @ 0x7f4a39dc34fb process::subprocess() >>>>> @ 0x43cc9c >>>>> mesos::internal::docker::DockerExecutorProcess::launchHealthCheck() >>>>> @ 0x7f4a39d924f4 process::ProcessManager::resume() >>>>> @ 0x7f4a39d92827 >>>>> _ZNSt6thread5_ImplISt12_Bind_simpleIFSt5_BindIFZN7process14ProcessManager12init_threadsEvEUlRKSt11atomic_boolE_St17reference_wrapperIS6_EEEvEEE6_M_runEv >>>>> @ 0x7f4a38a47e40 (unknown) >>>>> @ 0x7f4a3825d182 start_thread >>>>> @ 0x7f4a37f8a47d (unknown) >>>> >>>> >>>> Do any of you know which host the path >>>> "/tmp/mesos/slaves/16b49e90-6852-4c91-8e70-d89c54f25668-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.b89f2ceb-6d62-11e5-9827-080027477de0/runs/73dbfe88-1dbb-4f61-9a52-c365558cdbfc/mesos-health-check" >>>> should exist on? It definitely doesn't exist on the slave, hence >>>> execution failing. >>>> >>>> This is with current master, git hash >>>> 5058fac1083dc91bca54d33c26c810c17ad95dd1. >>>> >>>> commit 5058fac1083dc91bca54d33c26c810c17ad95dd1 >>>>> Author: Anand Mazumdar <mazumdar.an...@gmail.com> >>>>> Date: Tue Oct 6 17:37:41 2015 -0700 >>>> >>>> >>>> -Jay >>>> >>>> On Wed, Oct 7, 2015 at 5:23 PM, Jay Taylor <outtat...@gmail.com> wrote: >>>> >>>>> Update: >>>>> >>>>> I used https://github.com/deric/mesos-deb-packaging to compile and >>>>> package the latest master (0.26.x) and deployed it to the cluster, and now >>>>> health checks are working as advertised in both Marathon and my own >>>>> framework! Not sure what was going on with health-checks in 0.24.0.. >>>>> >>>>> Anyways, thanks again for your help Haosdent! >>>>> >>>>> Cheers, >>>>> Jay >>>>> >>>>> On Wed, Oct 7, 2015 at 12:53 PM, Jay Taylor <outtat...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Haosdent, >>>>>> >>>>>> Can you share your Marathon POST request that results in Mesos >>>>>> executing the health checks? >>>>>> >>>>>> Since we can reference the Marathon framework, I've been doing some >>>>>> digging around. >>>>>> >>>>>> Here are the details of my setup and findings: >>>>>> >>>>>> I put a few small hacks in Marathon: >>>>>> >>>>>> (1) Added com.googlecode.protobuf.format to Marathon's dependencies >>>>>> >>>>>> (2) Edited the following files so TaskInfo is dumped as JSON to >>>>>> /tmp/X in both the TaskFactory as well an right before the task is sent >>>>>> to >>>>>> Mesos via driver.launchTasks: >>>>>> >>>>>> src/main/scala/mesosphere/marathon/tasks/DefaultTaskFactory.scala: >>>>>> >>>>>> $ git diff >>>>>>> src/main/scala/mesosphere/marathon/tasks/DefaultTaskFactory.scala >>>>>>> @@ -25,6 +25,12 @@ class DefaultTaskFactory @Inject() ( >>>>>>> >>>>>>> new TaskBuilder(app, taskIdUtil.newTaskId, >>>>>>> config).buildIfMatches(offer, runningTasks).map { >>>>>>> case (taskInfo, ports) => >>>>>>> + import com.googlecode.protobuf.format.JsonFormat >>>>>>> + import java.io._ >>>>>>> + val bw = new BufferedWriter(new FileWriter(new >>>>>>> File("/tmp/taskjson1-" + taskInfo.getTaskId.getValue))) >>>>>>> + bw.write(JsonFormat.printToString(taskInfo)) >>>>>>> + bw.write("\n") >>>>>>> + bw.close() >>>>>>> CreatedTask( >>>>>>> taskInfo, >>>>>>> MarathonTasks.makeTask( >>>>>> >>>>>> >>>>>> >>>>>> src/main/scala/mesosphere/marathon/core/launcher/impl/TaskLauncherImpl.scala: >>>>>> >>>>>> $ git diff >>>>>>> src/main/scala/mesosphere/marathon/core/launcher/impl/TaskLauncherImpl.scala >>>>>>> @@ -24,6 +24,16 @@ private[launcher] class TaskLauncherImpl( >>>>>>> override def launchTasks(offerID: OfferID, taskInfos: >>>>>>> Seq[TaskInfo]): Boolean = { >>>>>>> val launched = withDriver(s"launchTasks($offerID)") { driver => >>>>>>> import scala.collection.JavaConverters._ >>>>>>> + var i = 0 >>>>>>> + for (i <- 0 to taskInfos.length - 1) { >>>>>>> + import com.googlecode.protobuf.format.JsonFormat >>>>>>> + import java.io._ >>>>>>> + val file = new File("/tmp/taskJson2-" + i.toString() + "-" >>>>>>> + taskInfos(i).getTaskId.getValue) >>>>>>> + val bw = new BufferedWriter(new FileWriter(file)) >>>>>>> + bw.write(JsonFormat.printToString(taskInfos(i))) >>>>>>> + bw.write("\n") >>>>>>> + bw.close() >>>>>>> + } >>>>>>> driver.launchTasks(Collections.singleton(offerID), >>>>>>> taskInfos.asJava) >>>>>>> } >>>>>> >>>>>> >>>>>> Then I built and deployed the hacked Marathon and restarted the >>>>>> marathon service. >>>>>> >>>>>> Next I created the app via the Marathon API ("hello app" is a >>>>>> container with a simple hello-world ruby app running on 0.0.0.0:8000) >>>>>> >>>>>> curl http://mesos-primary1a:8080/v2/groups -XPOST -H'Content-Type: >>>>>>> application/json' -d' >>>>>>> { >>>>>>> "id": "/app-81-1-hello-app", >>>>>>> "apps": [ >>>>>>> { >>>>>>> "id": "/app-81-1-hello-app/web-v11", >>>>>>> "container": { >>>>>>> "type": "DOCKER", >>>>>>> "docker": { >>>>>>> "image": >>>>>>> "docker-services1a:5000/gig1/app-81-1-hello-app-1444240966", >>>>>>> "network": "BRIDGE", >>>>>>> "portMappings": [ >>>>>>> { >>>>>>> "containerPort": 8000, >>>>>>> "hostPort": 0, >>>>>>> "protocol": "tcp" >>>>>>> } >>>>>>> ] >>>>>>> } >>>>>>> }, >>>>>>> "env": { >>>>>>> >>>>>>> }, >>>>>>> "healthChecks": [ >>>>>>> { >>>>>>> "protocol": "COMMAND", >>>>>>> "command": {"value": "exit 1"}, >>>>>>> "gracePeriodSeconds": 10, >>>>>>> "intervalSeconds": 10, >>>>>>> "timeoutSeconds": 10, >>>>>>> "maxConsecutiveFailures": 3 >>>>>>> } >>>>>>> ], >>>>>>> "instances": 1, >>>>>>> "cpus": 1, >>>>>>> "mem": 512 >>>>>>> } >>>>>>> ] >>>>>>> } >>>>>> >>>>>> >>>>>> $ ls /tmp/ >>>>>>> >>>>>>> taskjson1-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>> >>>>>>> taskJson2-0-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>> >>>>>> >>>>>> Do they match? >>>>>> >>>>>> $ md5sum /tmp/task* >>>>>>> 1b5115997e78e2611654059249d99578 >>>>>>> >>>>>>> /tmp/taskjson1-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>> 1b5115997e78e2611654059249d99578 >>>>>>> >>>>>>> /tmp/taskJson2-0-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>> >>>>>> >>>>>> Yes, so I am confident this is the information being sent across the >>>>>> wire to Mesos. >>>>>> >>>>>> Do they contain any health-check information? >>>>>> >>>>>> $ cat >>>>>>> /tmp/taskjson1-app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>>> { >>>>>>> "name":"web-v11.app-81-1-hello-app", >>>>>>> "task_id":{ >>>>>>> >>>>>>> "value":"app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0" >>>>>>> }, >>>>>>> "slave_id":{ >>>>>>> "value":"20150924-210922-1608624320-5050-1792-S1" >>>>>>> }, >>>>>>> "resources":[ >>>>>>> { >>>>>>> "name":"cpus", >>>>>>> "type":"SCALAR", >>>>>>> "scalar":{ >>>>>>> "value":1.0 >>>>>>> }, >>>>>>> "role":"*" >>>>>>> }, >>>>>>> { >>>>>>> "name":"mem", >>>>>>> "type":"SCALAR", >>>>>>> "scalar":{ >>>>>>> "value":512.0 >>>>>>> }, >>>>>>> "role":"*" >>>>>>> }, >>>>>>> { >>>>>>> "name":"ports", >>>>>>> "type":"RANGES", >>>>>>> "ranges":{ >>>>>>> "range":[ >>>>>>> { >>>>>>> "begin":31641, >>>>>>> "end":31641 >>>>>>> } >>>>>>> ] >>>>>>> }, >>>>>>> "role":"*" >>>>>>> } >>>>>>> ], >>>>>>> "command":{ >>>>>>> "environment":{ >>>>>>> "variables":[ >>>>>>> { >>>>>>> "name":"PORT_8000", >>>>>>> "value":"31641" >>>>>>> }, >>>>>>> { >>>>>>> "name":"MARATHON_APP_VERSION", >>>>>>> "value":"2015-10-07T19:35:08.386Z" >>>>>>> }, >>>>>>> { >>>>>>> "name":"HOST", >>>>>>> "value":"mesos-worker1a" >>>>>>> }, >>>>>>> { >>>>>>> "name":"MARATHON_APP_DOCKER_IMAGE", >>>>>>> >>>>>>> "value":"docker-services1a:5000/gig1/app-81-1-hello-app-1444240966" >>>>>>> }, >>>>>>> { >>>>>>> "name":"MESOS_TASK_ID", >>>>>>> >>>>>>> "value":"app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0" >>>>>>> }, >>>>>>> { >>>>>>> "name":"PORT", >>>>>>> "value":"31641" >>>>>>> }, >>>>>>> { >>>>>>> "name":"PORTS", >>>>>>> "value":"31641" >>>>>>> }, >>>>>>> { >>>>>>> "name":"MARATHON_APP_ID", >>>>>>> "value":"/app-81-1-hello-app/web-v11" >>>>>>> }, >>>>>>> { >>>>>>> "name":"PORT0", >>>>>>> "value":"31641" >>>>>>> } >>>>>>> ] >>>>>>> }, >>>>>>> "shell":false >>>>>>> }, >>>>>>> "container":{ >>>>>>> "type":"DOCKER", >>>>>>> "docker":{ >>>>>>> >>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-1444240966", >>>>>>> "network":"BRIDGE", >>>>>>> "port_mappings":[ >>>>>>> { >>>>>>> "host_port":31641, >>>>>>> "container_port":8000, >>>>>>> "protocol":"tcp" >>>>>>> } >>>>>>> ], >>>>>>> "privileged":false, >>>>>>> "force_pull_image":false >>>>>>> } >>>>>>> } >>>>>>> } >>>>>> >>>>>> >>>>>> No, I don't see anything about any health check. >>>>>> >>>>>> Mesos STDOUT for the launched task: >>>>>> >>>>>> --container="mesos-20150924-210922-1608624320-5050-1792-S1.14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>> --docker="docker" --help="false" --initialize_driver_logging="true" >>>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>> --sandbox_directory="/tmp/mesos/slaves/20150924-210922-1608624320-5050-1792-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0/runs/14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>> --stop_timeout="0ns" >>>>>>> --container="mesos-20150924-210922-1608624320-5050-1792-S1.14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>> --docker="docker" --help="false" --initialize_driver_logging="true" >>>>>>> --logbufsecs="0" --logging_level="INFO" >>>>>>> --mapped_directory="/mnt/mesos/sandbox" --quiet="false" >>>>>>> --sandbox_directory="/tmp/mesos/slaves/20150924-210922-1608624320-5050-1792-S1/frameworks/20150821-214332-1407297728-5050-18973-0000/executors/app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0/runs/14335f1f-3774-4862-a55b-e9c76cd0f2da" >>>>>>> --stop_timeout="0ns" >>>>>>> Registered docker executor on mesos-worker1a >>>>>>> Starting task >>>>>>> app-81-1-hello-app_web-v11.84c0f441-6d2a-11e5-98ba-080027477de0 >>>>>> >>>>>> >>>>>> And STDERR: >>>>>> >>>>>> I1007 19:35:08.790743 4612 exec.cpp:134] Version: 0.24.0 >>>>>>> I1007 19:35:08.793416 4619 exec.cpp:208] Executor registered on >>>>>>> slave 20150924-210922-1608624320-5050-1792-S1 >>>>>>> WARNING: Your kernel does not support swap limit capabilities, >>>>>>> memory limited without swap. >>>>>> >>>>>> >>>>>> Again, nothing about any health checks. >>>>>> >>>>>> Any ideas of other things to try or what I could be missing? Can't >>>>>> say either way about the Mesos health-check system working or not if >>>>>> Marathon won't put the health-check into the task it sends to Mesos. >>>>>> >>>>>> Thanks for all your help! >>>>>> >>>>>> Best, >>>>>> Jay >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>> On Tue, Oct 6, 2015 at 11:24 PM, haosdent <haosd...@gmail.com> wrote: >>>>>> >>>>>>> Maybe you could post your executor stdout/stderr so that we could >>>>>>> know whether health check running not. >>>>>>> >>>>>>> On Wed, Oct 7, 2015 at 2:15 PM, haosdent <haosd...@gmail.com> wrote: >>>>>>> >>>>>>>> marathon also use mesos health check. When I use health check, I >>>>>>>> could saw the log like this in executor stdout. >>>>>>>> >>>>>>>> ``` >>>>>>>> Registered docker executor on xxxxx >>>>>>>> Starting task test-health-check.822a5fd2-6cba-11e5-b5ce-0a0027000000 >>>>>>>> Launching health check process: >>>>>>>> /home/haosdent/mesos/build/src/.libs/mesos-health-check --executor=xxxx >>>>>>>> Health check process launched at pid: 9895 >>>>>>>> Received task health update, healthy: true >>>>>>>> ``` >>>>>>>> >>>>>>>> On Wed, Oct 7, 2015 at 12:51 PM, Jay Taylor <outtat...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> I am using my own framework, and the full task info I'm using is >>>>>>>>> posted earlier in this thread. Do you happen to know if Marathon uses >>>>>>>>> Mesos's health checks for its health check system? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Oct 6, 2015, at 9:01 PM, haosdent <haosd...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> Yes, launch the health task through its definition in taskinfo. Do >>>>>>>>> you launch your task through Marathon? I could test it in my side. >>>>>>>>> >>>>>>>>> On Wed, Oct 7, 2015 at 11:56 AM, Jay Taylor <outtat...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Precisely, and there are none of those statements. Are you or >>>>>>>>>> others confident health-checks are part of the code path when >>>>>>>>>> defined via >>>>>>>>>> task info for docker container tasks? Going through the code, I >>>>>>>>>> wasn't >>>>>>>>>> able to find the linkage for anything other than health-checks >>>>>>>>>> triggered >>>>>>>>>> through a custom executor. >>>>>>>>>> >>>>>>>>>> With that being said it is a pretty good sized code base and I'm >>>>>>>>>> not very familiar with it, so my analysis this far has by no means >>>>>>>>>> been >>>>>>>>>> exhaustive. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Oct 6, 2015, at 8:41 PM, haosdent <haosd...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>> When health check launch, it would have a log like this in your >>>>>>>>>> executor stdout >>>>>>>>>> ``` >>>>>>>>>> Health check process launched at pid xxx >>>>>>>>>> ``` >>>>>>>>>> >>>>>>>>>> On Wed, Oct 7, 2015 at 11:37 AM, Jay Taylor <outtat...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> I'm happy to try this, however wouldn't there be output in the >>>>>>>>>>> logs with the string "health" or "Health" if the health-check were >>>>>>>>>>> active? >>>>>>>>>>> None of my master or slave logs contain the string.. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Oct 6, 2015, at 7:45 PM, haosdent <haosd...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>> Could you use "exit 1" instead of "sleep 5" to see whether could >>>>>>>>>>> see unhealthy status in your task stdout/stderr. >>>>>>>>>>> >>>>>>>>>>> On Wed, Oct 7, 2015 at 10:38 AM, Jay Taylor <outtat...@gmail.com >>>>>>>>>>> > wrote: >>>>>>>>>>> >>>>>>>>>>>> My current version is 0.24.1. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Oct 6, 2015 at 7:30 PM, haosdent <haosd...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> yes, adam also help commit it to 0.23.1 and 0.24.1 >>>>>>>>>>>>> https://github.com/apache/mesos/commit/8c0ed92de3925d4312429bfba01b9b1ccbcbbef0 >>>>>>>>>>>>> >>>>>>>>>>>>> https://github.com/apache/mesos/commit/09e367cd69aa39c156c9326d44f4a7b829ba3db7 >>>>>>>>>>>>> Are you use one of this version? >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Oct 7, 2015 at 10:26 AM, haosdent <haosd...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> I remember 0.23.1 and 0.24.1 contains this backport, let me >>>>>>>>>>>>>> double check. >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Oct 7, 2015 at 10:01 AM, Jay Taylor < >>>>>>>>>>>>>> outtat...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Oops- Now I see you already said it's in master. I'll look >>>>>>>>>>>>>>> there :) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks again! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 6:59 PM, Jay Taylor < >>>>>>>>>>>>>>> j...@jaytaylor.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Great, thanks for the quick reply Tim! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Do you know if there is a branch I can checkout to test it >>>>>>>>>>>>>>>> out? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Oct 6, 2015 at 6:54 PM, Timothy Chen < >>>>>>>>>>>>>>>> t...@mesosphere.io> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Jay, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> We just added health check support for docker tasks that's >>>>>>>>>>>>>>>>> in master but not yet released. It will run docker exec with >>>>>>>>>>>>>>>>> the command >>>>>>>>>>>>>>>>> you provided as health checks. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It should be in the next release. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Tim >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Oct 6, 2015, at 6:49 PM, Jay Taylor < >>>>>>>>>>>>>>>>> outtat...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Does Mesos support health checks for docker image tasks? >>>>>>>>>>>>>>>>> Mesos seems to be ignoring the TaskInfo.HealthCheck field for >>>>>>>>>>>>>>>>> me. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Example TaskInfo JSON received back from Mesos: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "name":"hello-app.web.v3", >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "task_id":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> "value":"hello-app_web-v3.fc05a1a5-1e06-4e61-9879-be0d97cd3eec" >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "slave_id":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "value":"20150924-210922-1608624320-5050-1792-S1" >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "resources":[ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "name":"cpus", >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "type":0, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "scalar":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "value":0.1 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "name":"mem", >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "type":0, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "scalar":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "value":256 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "name":"ports", >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "type":1, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "ranges":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "range":[ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "begin":31002, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "end":31002 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ] >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ], >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "command":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "container":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> "image":"docker-services1a:5000/test/app-81-1-hello-app-103" >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "shell":false >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "container":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "type":1, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "docker":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> "image":"docker-services1a:5000/gig1/app-81-1-hello-app-103", >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "network":2, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "port_mappings":[ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "host_port":31002, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "container_port":8000, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "protocol":"tcp" >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> ], >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "privileged":false, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "parameters":[], >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "force_pull_image":false >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> }, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "health_check":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "delay_seconds":5, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "interval_seconds":10, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "timeout_seconds":10, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "consecutive_failures":3, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "grace_period_seconds":0, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "command":{ >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "shell":true, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "value":"sleep 5", >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> "user":"root" >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have searched all machines and containers to see if they >>>>>>>>>>>>>>>>> ever run the command (in this case `sleep 5`), but have not >>>>>>>>>>>>>>>>> found any >>>>>>>>>>>>>>>>> indication that it is being executed. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> In the mesos src code the health-checks are invoked from >>>>>>>>>>>>>>>>> src/launcher/executor.cpp CommandExecutorProcess::launchTask. >>>>>>>>>>>>>>>>> Does this >>>>>>>>>>>>>>>>> mean that health-checks are only supported for custom >>>>>>>>>>>>>>>>> executors and not for >>>>>>>>>>>>>>>>> docker tasks? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> What I am trying to accomplish is to have the 0/non-zero >>>>>>>>>>>>>>>>> exit-status of a health-check command translate to task >>>>>>>>>>>>>>>>> health. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> Jay >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> Best Regards, >>>>>>>>>>>>> Haosdent Huang >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Best Regards, >>>>>>>>>>> Haosdent Huang >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Best Regards, >>>>>>>>>> Haosdent Huang >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best Regards, >>>>>>>>> Haosdent Huang >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Best Regards, >>>>>>>> Haosdent Huang >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best Regards, >>>>>>> Haosdent Huang >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Best Regards, >>> Haosdent Huang >>> >> >> > > > -- > Best Regards, > Haosdent Huang > > -- Best Regards, Haosdent Huang
