[ https://issues.apache.org/jira/browse/MESOS-6952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830436#comment-15830436 ]
Sathish Kumar edited comment on MESOS-6952 at 1/19/17 9:59 PM: --------------------------------------------------------------- In below log we can see that after 14:42:17. its staged and until we restarted around 20.53 it was in staging state. FYI no slave reboot/leader election happened Please find the attached master logs {noformat} I0119 14:41:13.023109 29504 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:13.023146 29504 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:14.037518 29508 master.cpp:4763] Status update TASK_RUNNING (UUID: a3b53759-3c7e-408c-aec9-80b048e38938) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:53.829838 29508 master.cpp:4763] Status update TASK_FAILED (UUID: c65c736c-a00b-4ef3-beb6-589793169edb) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:53.893996 29499 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:54.736646 29495 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:54.736698 29495 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:55.708216 29509 master.cpp:4763] Status update TASK_RUNNING (UUID: 46408395-d5f5-4db2-babf-cefc2145f7f4) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.185272 29494 master.cpp:4763] Status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.230947 29494 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.862722 29500 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.862761 29500 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:16.312088 29504 master.cpp:4763] Status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:17.093446 29504 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:17.101451 29498 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:17.101490 29498 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) W0119 20:53:00.665473 29506 master.cpp:5592] Task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) during re-registration : reconciling with the slave W0119 20:53:00.672590 29506 master.cpp:5592] Task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) during re-registration : reconciling with the slave I0119 20:53:02.200562 29506 master.cpp:4763] Status update TASK_LOST (UUID: a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 20:53:02.200742 29506 master.cpp:4763] Status update TASK_LOST (UUID: a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 20:53:02.647492 29506 master.cpp:4763] Status update TASK_LOST (UUID: a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 20:53:02.663074 29506 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) {noformat} was (Author: sathee005): In below log we can see that after 14:42:17. its staged and until we restarted around 20.53 it was in staging state. FYI no slave reboot/leader election happened Please find the attached master logs \{noformat\} I0119 14:41:13.023109 29504 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:13.023146 29504 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:14.037518 29508 master.cpp:4763] Status update TASK_RUNNING (UUID: a3b53759-3c7e-408c-aec9-80b048e38938) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:53.829838 29508 master.cpp:4763] Status update TASK_FAILED (UUID: c65c736c-a00b-4ef3-beb6-589793169edb) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:53.893996 29499 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:54.736646 29495 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:54.736698 29495 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:41:55.708216 29509 master.cpp:4763] Status update TASK_RUNNING (UUID: 46408395-d5f5-4db2-babf-cefc2145f7f4) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.185272 29494 master.cpp:4763] Status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.230947 29494 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.862722 29500 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:15.862761 29500 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:16.312088 29504 master.cpp:4763] Status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:17.093446 29504 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:17.101451 29498 master.hpp:177] Adding task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 (distancematrix8.prod-foo-dcos.foobar.net) I0119 14:42:17.101490 29498 master.cpp:3589] Launching task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 (chronos-2.4.0) at scheduler-ebdbe062-3bc9-4683-94ce-96d7003a7fcc@10.14.23.221:55368 with resources cpus(*):0.4; mem(*):1024; disk(*):256 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) W0119 20:53:00.665473 29506 master.cpp:5592] Task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) during re-registration : reconciling with the slave W0119 20:53:00.672590 29506 master.cpp:5592] Task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 unknown to the slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) during re-registration : reconciling with the slave I0119 20:53:02.200562 29506 master.cpp:4763] Status update TASK_LOST (UUID: a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 20:53:02.200742 29506 master.cpp:4763] Status update TASK_LOST (UUID: a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 20:53:02.647492 29506 master.cpp:4763] Status update TASK_LOST (UUID: a66eeb61-258e-46b2-aeb7-b07093b2e1a6) for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) I0119 20:53:02.663074 29506 master.cpp:6487] Removing task ct:1484816820000:0:foocare_zendesk_round_robin: with resources cpus(*):0.4; mem(*):1024; disk(*):256 of framework 19393553-2061-4d2f-8c05-a0ba688334f4-0001 on slave 22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56 at slave(1)@10.14.38.239:5051 (distancematrix8.prod-foo-dcos.foobar.net) \{noformat\} > Mesos task state was stuck in staging even after executor terminated > -------------------------------------------------------------------- > > Key: MESOS-6952 > URL: https://issues.apache.org/jira/browse/MESOS-6952 > Project: Mesos > Issue Type: Bug > Components: executor > Affects Versions: 0.28.2 > Environment: ubuntu 14.04 > Reporter: Sathish Kumar > > Task is stuck at staging almost 6hours in stage even after slave executor is > terminated. > Mesos master keeps the task state in staging state. Since the task is stuck > at staging framework have not got the update from mesos-master > The issue got fixed after slave restart. > I can see in the slave logs Asked to run task ' which is > terminating/terminated > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for > status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for > task ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097193 107774 slave.cpp:1361] Got assigned task > ct:1484816820000:0:foocare_zendesk_round_robin: for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097453 107774 slave.cpp:1480] Launching task > ct:1484816820000:0:foocare_zendesk_round_robin: for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119 > 14:42:17.097527 107774 slave.cpp:1673] Asked to run task > 'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' which is > terminating/terminated > full Log of slave > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.066277 107763 slave.cpp:3012] Handling status update TASK_FAILED > (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from executor(1)@10.14.38.239:43937 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.134692 107766 status_update_manager.cpp:320] Received status update > TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.134753 107766 status_update_manager.cpp:824] Checkpointing UPDATE > for status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) > for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.142010 107767 slave.cpp:3410] Forwarding the update TASK_FAILED > (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.142119 107767 slave.cpp:3320] Sending acknowledgement for status > update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 to executor(1)@10.14.38.239:43937 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.226682 107761 status_update_manager.cpp:392] Received status update > acknowledgement (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.226759 107761 status_update_manager.cpp:824] Checkpointing ACK for > status update TASK_FAILED (UUID: 5e4147e8-f11c-4950-ba7b-c4e7f8bc5932) for > task ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.858510 107759 slave.cpp:1361] Got assigned task > ct:1484816820000:0:foocare_zendesk_round_robin: for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.858762 107759 slave.cpp:1480] Launching task > ct:1484816820000:0:foocare_zendesk_round_robin: for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.859004 107759 slave.cpp:1711] Queuing task > 'ct:1484816820000:0:foocare_zendesk_round_robin:' for executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:15.939483 107759 slave.cpp:1863] Sending queued task > 'ct:1484816820000:0:foocare_zendesk_round_robin:' to executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:16.141394 107762 slave.cpp:3871] Executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 exited with status 0 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:16.141451 107762 slave.cpp:3012] Handling status update TASK_FAILED > (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:16.141849 107762 status_update_manager.cpp:320] Received status update > TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:16.141989 107762 status_update_manager.cpp:824] Checkpointing UPDATE > for status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) > for task ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:16.147343 107766 slave.cpp:3410] Forwarding the update TASK_FAILED > (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 to master@10.14.23.181:5050 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.089175 107759 status_update_manager.cpp:392] Received status update > acknowledgement (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.089251 107759 status_update_manager.cpp:824] Checkpointing ACK for > status update TASK_FAILED (UUID: 247bbeed-1d60-4d33-ac1e-9282266c54ee) for > task ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097193 107774 slave.cpp:1361] Got assigned task > ct:1484816820000:0:foocare_zendesk_round_robin: for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097453 107774 slave.cpp:1480] Launching task > ct:1484816820000:0:foocare_zendesk_round_robin: for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:W0119 > 14:42:17.097527 107774 slave.cpp:1673] Asked to run task > 'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' which is > terminating/terminated > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097568 107774 slave.cpp:3012] Handling status update TASK_LOST > (UUID: b999fb64-34f0-496d-be19-f5a7f998230e) for task > ct:1484816820000:0:foocare_zendesk_round_robin: of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 from @0.0.0.0:0 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097633 107774 slave.cpp:3975] Cleaning up executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' of framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 at executor(1)@10.14.38.239:43937 > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097790 107772 gc.cpp:55] Scheduling > '/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b' > for gc 6.99999886874074days in the future > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097836 107772 gc.cpp:55] Scheduling > '/data/mesos/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:' > for gc 6.99999886832296days in the future > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097869 107772 gc.cpp:55] Scheduling > '/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:/runs/6b8922ff-3f57-42a0-97d1-d79c1de3d93b' > for gc 6.99999886819259days in the future > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.INFO.20161004-154315.107733:I0119 > 14:42:17.097888 107772 gc.cpp:55] Scheduling > '/data/mesos/meta/slaves/22c4f06b-d107-4cf4-86b1-81a6cce5441a-S56/frameworks/19393553-2061-4d2f-8c05-a0ba688334f4-0001/executors/ct:1484816820000:0:foocare_zendesk_round_robin:' > for gc 6.99999886809185days in the future > mesos-slave.distancematrix8.prod-foo-dcos.foobar.net.invalid-user.log.WARNING.20161004-154318.107733:W0119 > 14:42:17.097527 107774 slave.cpp:1673] Asked to run task > 'ct:1484816820000:0:foocare_zendesk_round_robin:' for framework > 19393553-2061-4d2f-8c05-a0ba688334f4-0001 with executor > 'ct:1484816820000:0:foocare_zendesk_round_robin:' which is > terminating/terminated -- This message was sent by Atlassian JIRA (v6.3.4#6332)