Tan N. Le created MESOS-10233:
---------------------------------
Summary: containers were not cleaned up properly and left running.
Key: MESOS-10233
URL: https://issues.apache.org/jira/browse/MESOS-10233
Project: Mesos
Issue Type: Task
Environment: aurora-scheduler 0.25.0
mesos 1.11.0
executor plugin: DCE [https://github.com/paypal/dce-go] based on mesos-go v0.002
Reporter: Tan N. Le
We observe that tasks were in STARTING and mesos tried to killed and cleaned
them up dueo to OOM.
however, cgroup freezer files are not there and it assumes the containers are
being cleaned.
the containers left running but the tasks reported lost in aurora/mesos.
================
aurora logs
I1026 05:16:55.886 [TaskEventBatchWorker, StateMachine]
-b0b685ca-3ded-4304-a591-9241d06d7728 state machine transition INIT -> PENDING
I1026 05:16:55.984 [TaskGroupBatchWorker, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
state machine transition PENDING -> ASSIGNED
I1026 05:16:55.984 [TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent
gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com (id
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307) is being assigned task for
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728.
I1026 05:16:57.402 [Thread-1715969,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in
state TASK_STARTING from SOURCE_EXECUTOR
I1026 05:16:57.402 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
state machine transition ASSIGNED -> STARTING
W1026 05:18:03.376 [Thread-1717148,
MesosCallbackHandler$MesosCallbackHandlerImpl] Lost executor
compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
on slave a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 with status -1
I1026 05:18:03.377 [Thread-1717149,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in
state TASK_FAILED from SOURCE_AGENT with REASON_EXECUTOR_TERMINATED: Abnormal
executor termination: Failed to kill all processes in the container: Timed out
after 1mins
I1026 05:18:03.390 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
state machine transition STARTING -> FAILED
I1026 05:18:03.390 [TaskStatusHandlerImpl, StateManagerImpl] Task being
rescheduled:
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
I1026 05:21:08.948 [Thread-1720928,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 in
state TASK_RUNNING from SOURCE_EXECUTOR
I1026 05:21:08.949 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
state machine transition FAILED -> RUNNING (not allowed)
I1026 05:21:08.950 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
state machine transition FAILED -> LOST (not allowed)
=================
mesos-master logs
I1026 05:16:55.991168 29973 master.cpp:3873] Adding executor
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728'
with resources cpus(allocated: aurora):0.1; mem(allocated: aurora):256 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:16:55.991324 29973 master.cpp:3899] Adding task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
with resources cpus(allocated: aurora):0.9; disk(allocated: aurora):100;
mem(allocated: aurora):4096; ports(allocated: aurora):[10020-10020,
10076-10076, 10137-10137, 10139-10139, 10150-10150] of framework
9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:16:56.090255 29973 master.cpp:5035] Launching task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 with
resources
[\{"allocation_info":{"role":"aurora"},"name":"cpus","scalar":\{"value":0.9},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"disk","scalar":\{"value":100.0},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"mem","scalar":\{"value":4096.0},"type":"SCALAR"},\{"allocation_info":{"role":"aurora"},"name":"ports","ranges":\{"range":[{"begin":10020,"end":10020},\{"begin":10076,"end":10076},\{"begin":10137,"end":10137},\{"begin":10139,"end":10139},\{"begin":10150,"end":10150}]},"type":"RANGES"}]
on agent a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at
slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com) on new executor
I1026 05:16:57.401727 29979 master.cpp:8160] Status update TASK_STARTING
(Status UUID: 9d316a48-3656-11ec-8d10-42010ab432d2) for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:16:57.401800 29979 master.cpp:8217] Forwarding status update
TASK_STARTING (Status UUID: 9d316a48-3656-11ec-8d10-42010ab432d2) for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000
I1026 05:16:57.401979 29979 master.cpp:11195] Updating the state of task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (latest state:
TASK_STARTING, status update state: TASK_STARTING)
I1026 05:16:57.409750 29990 master.cpp:5870] Processing ACKNOWLEDGE call for
status 9d316a48-3656-11ec-8d10-42010ab432d2 for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307
I1026 05:18:03.375757 29982 master.cpp:8497] Executor
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728'
of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com): wait status -1
I1026 05:18:03.375849 29982 master.cpp:11335] Removing executor
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728'
with resources cpus(allocated: aurora):0.1; mem(allocated: aurora):256 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:18:03.376816 29990 master.cpp:8160] Status update TASK_FAILED (Status
UUID: 7fbe4d73-2eed-4713-8774-8299f3df9736) for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:18:03.376888 29990 master.cpp:8217] Forwarding status update
TASK_FAILED (Status UUID: 7fbe4d73-2eed-4713-8774-8299f3df9736) for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000
I1026 05:18:03.377179 29990 master.cpp:11195] Updating the state of task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (latest state: TASK_FAILED,
status update state: TASK_FAILED)
I1026 05:18:03.505921 29974 master.cpp:5870] Processing ACKNOWLEDGE call for
status 7fbe4d73-2eed-4713-8774-8299f3df9736 for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307
I1026 05:18:03.505990 29974 master.cpp:11295] Removing task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728
with resources cpus(allocated: aurora):0.9; disk(allocated: aurora):100;
mem(allocated: aurora):4096; ports(allocated: aurora):[10020-10020,
10076-10076, 10137-10137, 10139-10139, 10150-10150] of framework
9f48d831-63e7-4556-86ab-463a69389e4d-0000 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:21:08.947870 29968 master.cpp:8160] Status update TASK_RUNNING (Status
UUID: 332069b9-3657-11ec-8d10-42010ab432d2) for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:21:08.947954 29968 master.cpp:8217] Forwarding status update
TASK_RUNNING (Status UUID: 332069b9-3657-11ec-8d10-42010ab432d2) for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000
W1026 05:21:08.948144 29968 master.cpp:8190] Could not lookup task for status
update TASK_RUNNING (Status UUID: 332069b9-3657-11ec-8d10-42010ab432d2) for
task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 from agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307 at slave(1)@10.180.50.210:5051
(gpma771518.gpf-prod.us-central1.gcp.dev.paypalinc.com)
I1026 05:21:08.949649 29980 master.cpp:5696] Processing KILL call for task
'mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728'
of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083
W1026 05:21:08.949703 29980 master.cpp:5703] Cannot kill task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 because it is
unknown; performing reconciliation
I1026 05:21:08.949753 29980 master.cpp:9140] Sending explicit reconciliation
state TASK_LOST for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083
I1026 05:21:08.949921 29980 master.cpp:5870] Processing ACKNOWLEDGE call for
status 332069b9-3657-11ec-8d10-42010ab432d2 for task
mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728 of
framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 (Aurora) at
[email protected]:8083 on agent
a76961ab-bba0-46e5-ae7b-b234057b7a33-S307
==================
mesos-slave logs
E1026 05:17:03.372925 849 memory.cpp:665] Failed to read
'memory.limit_in_bytes': No such file or directory
E1026 05:17:03.373068 849 memory.cpp:674] Failed to read
'memory.max_usage_in_bytes': No such file or directory
E1026 05:17:03.373138 849 memory.cpp:686] Failed to read 'memory.stat': No such
file or directory
W1026 05:17:03.373395 849 linux_launcher.cpp:600] Couldn't find freezer cgroup
for container f9e69838-169e-4837-be5e-11ef4860bb01 so assuming partially
destroyed
W1026 05:17:03.373773 848 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.375150 839 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.377418 841 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.381808 832 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.390095 829 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.406415 830 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.438736 843 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.503955 836 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.632373 846 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:03.888777 845 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:04.401167 826 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:05.425781 838 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:07.475064 827 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:11.571998 847 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:19.764526 847 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
W1026 05:17:36.149070 841 cgroups.cpp:294] Removal of cgroup
/sys/fs/cgroup/systemd/mesos/f9e69838-169e-4837-be5e-11ef4860bb01 failed with
EBUSY, will try again
E1026 05:18:03.375078 835 slave.cpp:7085] Termination of executor
'compose-mstestenv-msmaster4int-g-mppnodeweb-a-3-b0b685ca-3ded-4304-a591-9241d06d7728'
of framework 9f48d831-63e7-4556-86ab-463a69389e4d-0000 failed: Failed to kill
all processes in the container: Timed out after 1mins
I1026 05:16:39.505 [qtp1387308322-1609115, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92
state machine transition RUNNING -> KILLING
I1026 05:16:52.192 [Thread-1715877,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92 in
state TASK_KILLED from SOURCE_EXECUTOR
I1026 05:16:52.192 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-36cae1b6-0b73-4caf-9820-04af86396a92
state machine transition KILLING -> KILLED
I1026 05:16:52.243 [TaskEventBatchWorker, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
state machine transition INIT -> PENDING
I1026 05:16:52.300 [TaskGroupBatchWorker, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
state machine transition PENDING -> ASSIGNED
I1026 05:16:52.300 [TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent
gpma773817.gpf-prod.us-central1.gcp.dev.paypalinc.com (id
a76961ab-bba0-46e5-ae7b-b234057b7a33-S470) is being assigned task for
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649.
I1026 05:16:53.643 [Thread-1715899,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in
state TASK_STARTING from SOURCE_EXECUTOR
I1026 05:16:53.643 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
state machine transition ASSIGNED -> STARTING
W1026 05:18:02.991 [Thread-1717141,
MesosCallbackHandler$MesosCallbackHandlerImpl] Lost executor
compose-mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
on slave a76961ab-bba0-46e5-ae7b-b234057b7a33-S470 with status -1
I1026 05:18:02.991 [Thread-1717142,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in
state TASK_FAILED from SOURCE_AGENT with REASON_EXECUTOR_TERMINATED: Abnormal
executor termination: Failed to kill all processes in the container: Timed out
after 1mins
I1026 05:18:03.094 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
state machine transition STARTING -> FAILED
I1026 05:18:03.095 [TaskStatusHandlerImpl, StateManagerImpl] Task being
rescheduled:
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
I1026 05:18:03.095 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889
state machine transition INIT -> PENDING
I1026 05:18:03.227 [TaskGroupBatchWorker, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889
state machine transition PENDING -> ASSIGNED
I1026 05:18:03.227 [TaskGroupBatchWorker, TaskAssignerImpl] Offer on agent
gpma778297.gpf-prod.us-central1.gcp.dev.paypalinc.com (id
a76961ab-bba0-46e5-ae7b-b234057b7a33-S609) is being assigned task for
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889.
I1026 05:18:04.520 [Thread-1717191,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 in
state TASK_STARTING from SOURCE_EXECUTOR
I1026 05:18:04.520 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889
state machine transition ASSIGNED -> STARTING
I1026 05:20:52.246 [Thread-1720630,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649 in
state TASK_RUNNING from SOURCE_EXECUTOR
I1026 05:20:52.246 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
state machine transition FAILED -> RUNNING (not allowed)
I1026 05:20:52.248 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-a324b802-ced3-4d2b-8ccb-41884fe89649
state machine transition FAILED -> LOST (not allowed)
I1026 05:22:50.010 [Thread-1723190,
MesosCallbackHandler$MesosCallbackHandlerImpl] Received status update for task
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889 in
state TASK_RUNNING from SOURCE_EXECUTOR
I1026 05:22:50.010 [TaskStatusHandlerImpl, StateMachine]
mstestenv-msmaster4int-g-mppnodeweb-a-6-1a6d12ba-ed47-4fb7-87b7-601b9bcdb889
state machine transition STARTING -> RUNNING
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
