[
https://issues.apache.org/jira/browse/MESOS-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sathish Kumar updated MESOS-6952:
---------------------------------
Description:
Task was stuck at staging state almost 6hours even after slave executor is
terminated on the slave. Since the task was stuck at staging, framework have
not received update from mesos-master.
The issue got fixed after slave restart and the task was moved from staging to
task lost state.
I can see in the slave logs Asked to run task ' which is terminating/terminated
{noformat}
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for
status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097193 107774 slave.cpp:1361] Got assigned task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097453 107774 slave.cpp:1480] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119
14:42:17.097527 107774 slave.cpp:1673] Asked to run task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
terminating/terminated
{noformat}
full Log of slave
{noformat}
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.066277 107763 slave.cpp:3012] Handling status update TASK_FAILED
(UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.134692 107766 status_update_manager.cpp:320] Received status update
TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.134753 107766 status_update_manager.cpp:824] Checkpointing UPDATE for
status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.142010 107767 slave.cpp:3410] Forwarding the update TASK_FAILED
(UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.142119 107767 slave.cpp:3320] Sending acknowledgement for status
update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 to executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.226682 107761 status_update_manager.cpp:392] Received status update
acknowledgement (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.226759 107761 status_update_manager.cpp:824] Checkpointing ACK for
status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.858510 107759 slave.cpp:1361] Got assigned task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.858762 107759 slave.cpp:1480] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.859004 107759 slave.cpp:1711] Queuing task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.939483 107759 slave.cpp:1863] Sending queued task
'ct:1484816820000:0:foocare_zendesk_round_robin:' to executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141394 107762 slave.cpp:3871] Executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 exited with status 0
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141451 107762 slave.cpp:3012] Handling status update TASK_FAILED
(UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141849 107762 status_update_manager.cpp:320] Received status update
TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141989 107762 status_update_manager.cpp:824] Checkpointing UPDATE for
status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.147343 107766 slave.cpp:3410] Forwarding the update TASK_FAILED
(UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.089175 107759 status_update_manager.cpp:392] Received status update
acknowledgement (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for
status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097193 107774 slave.cpp:1361] Got assigned task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097453 107774 slave.cpp:1480] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119
14:42:17.097527 107774 slave.cpp:1673] Asked to run task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
terminating/terminated
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097568 107774 slave.cpp:3012] Handling status update TASK_LOST (UUID:
b999fb64-34f0-496d-be19-f5a7f998230e) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097633 107774 slave.cpp:3975] Cleaning up executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097790 107772 gc.cpp:55] Scheduling
'/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b'
for gc 6.99999886874074days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097836 107772 gc.cpp:55] Scheduling
'/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:'
for gc 6.99999886832296days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097869 107772 gc.cpp:55] Scheduling
'/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b'
for gc 6.99999886819259days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097888 107772 gc.cpp:55] Scheduling
'/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:'
for gc 6.99999886809185days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.WARNING.20161004-154318.107733:W0119
14:42:17.097527 107774 slave.cpp:1673] Asked to run task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
terminating/terminated
{noformat}
Master logs
{noformat}
I0119 14:41:13.023109 29504 master.hpp:177] Adding task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:13.023146 29504 master.cpp:3589] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:14.037518 29508 master.cpp:4763] Status update TASK_RUNNING (UUID:
a3b53759-3c7e-408c-aec9-80b048e38938) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:53.829838 29508 master.cpp:4763] Status update TASK_FAILED (UUID:
c65c736c-a00b-4ef3-beb6-589793169edb) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:53.893996 29499 master.cpp:6487] Removing task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001
on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:54.736646 29495 master.hpp:177] Adding task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:54.736698 29495 master.cpp:3589] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:41:55.708216 29509 master.cpp:4763] Status update TASK_RUNNING (UUID:
46408395-d5f5-4db2-babf-cefc2145f7f4) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:15.185272 29494 master.cpp:4763] Status update TASK_FAILED (UUID:
5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:15.230947 29494 master.cpp:6487] Removing task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001
on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:15.862722 29500 master.hpp:177] Adding task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:15.862761 29500 master.cpp:3589] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:16.312088 29504 master.cpp:4763] Status update TASK_FAILED (UUID:
247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:17.093446 29504 master.cpp:6487] Removing task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001
on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:17.101451 29498 master.hpp:177] Adding task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 14:42:17.101490 29498 master.cpp:3589] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
W0119 20:53:00.665473 29506 master.cpp:5592] Task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net) during re-registration : reconciling
with the slave
W0119 20:53:00.672590 29506 master.cpp:5592] Task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net) during re-registration : reconciling
with the slave
I0119 20:53:02.200562 29506 master.cpp:4763] Status update TASK_LOST (UUID:
a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 20:53:02.200742 29506 master.cpp:4763] Status update TASK_LOST (UUID:
a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 20:53:02.647492 29506 master.cpp:4763] Status update TASK_LOST (UUID:
a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
I0119 20:53:02.663074 29506 master.cpp:6487] Removing task
ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001
on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
(distancematrix8.prod-foo-dcos.foobar.net)
{noformat}
was:
Task was stuck at staging state almost 6hours even after slave executor is
terminated on the slave. Since the task was stuck at staging, framework have
not received update from mesos-master.
The issue got fixed after slave restart and the task was moved from staging to
task lost state.
I can see in the slave logs Asked to run task ' which is terminating/terminated
{noformat}
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for
status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097193 107774 slave.cpp:1361] Got assigned task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097453 107774 slave.cpp:1480] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119
14:42:17.097527 107774 slave.cpp:1673] Asked to run task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
terminating/terminated
{noformat}
full Log of slave
{noformat}
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.066277 107763 slave.cpp:3012] Handling status update TASK_FAILED
(UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.134692 107766 status_update_manager.cpp:320] Received status update
TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.134753 107766 status_update_manager.cpp:824] Checkpointing UPDATE for
status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.142010 107767 slave.cpp:3410] Forwarding the update TASK_FAILED
(UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.142119 107767 slave.cpp:3320] Sending acknowledgement for status
update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 to executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.226682 107761 status_update_manager.cpp:392] Received status update
acknowledgement (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.226759 107761 status_update_manager.cpp:824] Checkpointing ACK for
status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.858510 107759 slave.cpp:1361] Got assigned task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.858762 107759 slave.cpp:1480] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.859004 107759 slave.cpp:1711] Queuing task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:15.939483 107759 slave.cpp:1863] Sending queued task
'ct:1484816820000:0:foocare_zendesk_round_robin:' to executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141394 107762 slave.cpp:3871] Executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 exited with status 0
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141451 107762 slave.cpp:3012] Handling status update TASK_FAILED
(UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141849 107762 status_update_manager.cpp:320] Received status update
TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.141989 107762 status_update_manager.cpp:824] Checkpointing UPDATE for
status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:16.147343 107766 slave.cpp:3410] Forwarding the update TASK_FAILED
(UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.089175 107759 status_update_manager.cpp:392] Received status update
acknowledgement (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for
status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097193 107774 slave.cpp:1361] Got assigned task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097453 107774 slave.cpp:1480] Launching task
ct:1484816820000:0:foocare_zendesk_round_robin: for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119
14:42:17.097527 107774 slave.cpp:1673] Asked to run task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
terminating/terminated
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097568 107774 slave.cpp:3012] Handling status update TASK_LOST (UUID:
b999fb64-34f0-496d-be19-f5a7f998230e) for task
ct:1484816820000:0:foocare_zendesk_round_robin: of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097633 107774 slave.cpp:3975] Cleaning up executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097790 107772 gc.cpp:55] Scheduling
'/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b'
for gc 6.99999886874074days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097836 107772 gc.cpp:55] Scheduling
'/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:'
for gc 6.99999886832296days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097869 107772 gc.cpp:55] Scheduling
'/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b'
for gc 6.99999886819259days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
14:42:17.097888 107772 gc.cpp:55] Scheduling
'/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:'
for gc 6.99999886809185days in the future
mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.WARNING.20161004-154318.107733:W0119
14:42:17.097527 107774 slave.cpp:1673] Asked to run task
'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
terminating/terminated
{noformat}
Master logs
{noformat}
> Mesos task state was stuck in staging even after executor terminated
> --------------------------------------------------------------------
>
> Key: MESOS-6952
> URL: https://issues.apache.org/jira/browse/MESOS-6952
> Project: Mesos
> Issue Type: Bug
> Components: executor
> Affects Versions: 0.28.2
> Environment: ubuntu 14.04
> Reporter: Sathish Kumar
>
> Task was stuck at staging state almost 6hours even after slave executor is
> terminated on the slave. Since the task was stuck at staging, framework have
> not received update from mesos-master.
> The issue got fixed after slave restart and the task was moved from staging
> to task lost state.
> I can see in the slave logs Asked to run task ' which is
> terminating/terminated
> {noformat}
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for
> status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for
> task ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097193 107774 slave.cpp:1361] Got assigned task
> ct:1484816820000:0:foocare_zendesk_round_robin: for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097453 107774 slave.cpp:1480] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119
> 14:42:17.097527 107774 slave.cpp:1673] Asked to run task
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
> terminating/terminated
> {noformat}
> full Log of slave
> {noformat}
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.066277 107763 slave.cpp:3012] Handling status update TASK_FAILED
> (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from executor(1)@10.14.38.239:43937
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.134692 107766 status_update_manager.cpp:320] Received status update
> TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.134753 107766 status_update_manager.cpp:824] Checkpointing UPDATE
> for status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932)
> for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.142010 107767 slave.cpp:3410] Forwarding the update TASK_FAILED
> (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.142119 107767 slave.cpp:3320] Sending acknowledgement for status
> update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 to executor(1)@10.14.38.239:43937
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.226682 107761 status_update_manager.cpp:392] Received status update
> acknowledgement (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.226759 107761 status_update_manager.cpp:824] Checkpointing ACK for
> status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for
> task ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.858510 107759 slave.cpp:1361] Got assigned task
> ct:1484816820000:0:foocare_zendesk_round_robin: for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.858762 107759 slave.cpp:1480] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.859004 107759 slave.cpp:1711] Queuing task
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' for executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:15.939483 107759 slave.cpp:1863] Sending queued task
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' to executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:16.141394 107762 slave.cpp:3871] Executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 exited with status 0
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:16.141451 107762 slave.cpp:3012] Handling status update TASK_FAILED
> (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:16.141849 107762 status_update_manager.cpp:320] Received status update
> TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:16.141989 107762 status_update_manager.cpp:824] Checkpointing UPDATE
> for status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee)
> for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:16.147343 107766 slave.cpp:3410] Forwarding the update TASK_FAILED
> (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.089175 107759 status_update_manager.cpp:392] Received status update
> acknowledgement (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for
> status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for
> task ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097193 107774 slave.cpp:1361] Got assigned task
> ct:1484816820000:0:foocare_zendesk_round_robin: for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097453 107774 slave.cpp:1480] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119
> 14:42:17.097527 107774 slave.cpp:1673] Asked to run task
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
> terminating/terminated
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097568 107774 slave.cpp:3012] Handling status update TASK_LOST
> (UUID: b999fb64-34f0-496d-be19-f5a7f998230e) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097633 107774 slave.cpp:3975] Cleaning up executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097790 107772 gc.cpp:55] Scheduling
> '/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b'
> for gc 6.99999886874074days in the future
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097836 107772 gc.cpp:55] Scheduling
> '/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:'
> for gc 6.99999886832296days in the future
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097869 107772 gc.cpp:55] Scheduling
> '/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b'
> for gc 6.99999886819259days in the future
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119
> 14:42:17.097888 107772 gc.cpp:55] Scheduling
> '/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:'
> for gc 6.99999886809185days in the future
> mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.WARNING.20161004-154318.107733:W0119
> 14:42:17.097527 107774 slave.cpp:1673] Asked to run task
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor
> 'ct:1484816820000:0:foocare_zendesk_round_robin:' which is
> terminating/terminated
> {noformat}
> Master logs
> {noformat}
> I0119 14:41:13.023109 29504 master.hpp:177] Adding task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:13.023146 29504 master.cpp:3589] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
> scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
> resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:14.037518 29508 master.cpp:4763] Status update TASK_RUNNING
> (UUID: a3b53759-3c7e-408c-aec9-80b048e38938) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:53.829838 29508 master.cpp:4763] Status update TASK_FAILED (UUID:
> c65c736c-a00b-4ef3-beb6-589793169edb) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:53.893996 29499 master.cpp:6487] Removing task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:54.736646 29495 master.hpp:177] Adding task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:54.736698 29495 master.cpp:3589] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
> scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
> resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:41:55.708216 29509 master.cpp:4763] Status update TASK_RUNNING
> (UUID: 46408395-d5f5-4db2-babf-cefc2145f7f4) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:15.185272 29494 master.cpp:4763] Status update TASK_FAILED (UUID:
> 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:15.230947 29494 master.cpp:6487] Removing task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:15.862722 29500 master.hpp:177] Adding task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:15.862761 29500 master.cpp:3589] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
> scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
> resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:16.312088 29504 master.cpp:4763] Status update TASK_FAILED (UUID:
> 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:17.093446 29504 master.cpp:6487] Removing task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:17.101451 29498 master.hpp:177] Adding task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 14:42:17.101490 29498 master.cpp:3589] Launching task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at
> scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with
> resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> W0119 20:53:00.665473 29506 master.cpp:5592] Task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net) during re-registration :
> reconciling with the slave
> W0119 20:53:00.672590 29506 master.cpp:5592] Task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net) during re-registration :
> reconciling with the slave
> I0119 20:53:02.200562 29506 master.cpp:4763] Status update TASK_LOST (UUID:
> a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 20:53:02.200742 29506 master.cpp:4763] Status update TASK_LOST (UUID:
> a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 20:53:02.647492 29506 master.cpp:4763] Status update TASK_LOST (UUID:
> a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task
> ct:1484816820000:0:foocare_zendesk_round_robin: of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> I0119 20:53:02.663074 29506 master.cpp:6487] Removing task
> ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4;
> mem(*):1024; disk(*):256 of framework
> 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave
> 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051
> (distancematrix8.prod-foo-dcos.foobar.net)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
