[
https://issues.apache.org/jira/browse/MESOS-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tomasz Janiszewski updated MESOS-5536:
--------------------------------------
Description:
I'm running Mesos 0.28.0. Mesos {{slave(1)/state}} endpoint returns some
completed executors not in frameworks.completed_executors but in
frameworks.executors. Alsa this executor presents in {{monitor/statistics}}
{code:JavaScript:title=slave(1)/state}
{
"attributes": {...},
"completed_frameworks": [],
"flags": {...},
"frameworks": [
{
"checkpoint": true,
"completed_executors": [...],
"executors": [
{
"queued_tasks": [],
"tasks": [],
"completed_tasks": [
{
"discovery": {...},
"executor_id": "",
"framework_id":
"f65b163c-0faf-441f-ac14-91739fa4394c-0000",
"id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
"labels": [...],
"name": "service",
"resources": {...},
"slave_id":
"ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13",
"state": "TASK_KILLED",
"statuses": []
}
],
"container": "ead42e63-ac92-4ad0-a99c-4af9c3fa5e31",
"directory": "...",
"id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
"name": "Command Executor (Task:
service.a3b609b8-27ec-11e6-8044-02c89eb9127e) (Command: sh -c 'cd
service...')",
"resources": {...},
"source": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e"
},
...
],
}
],
"git_sha": "961edbd82e691a619a4c171a7aadc9c32957fa73",
"git_tag": "0.28.0",
"version": "0.28.0",
...
}
{code}
{code:title="var/log/mesos/mesos-slave.INFO"}
13:33:19.479182 [slave.cpp:1361] Got assigned task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:19.482566 [slave.cpp:1480] Launching task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:19.483921 [paths.cpp:528] Trying to chown
'/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
to user 'mesosuser'
13:33:19.504173 [slave.cpp:5367] Launching executor
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 with resources cpus(*):0.1; mem(*):32
in work directory
'/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
13:33:19.505537 [containerizer.cpp:666] Starting container
'ead42e63-ac92-4ad0-a99c-4af9c3fa5e31' for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
'f65b163c-0faf-441f-ac14-91739fa4394c-0000'
13:33:19.505734 [slave.cpp:1698] Queuing task
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
...
13:33:19.977483 [containerizer.cpp:1118] Checkpointing executor's forked pid
25576 to
'/tmp/mesos/meta/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31/pids/forked.pid'
13:33:35.775195 [slave.cpp:1891] Asked to kill task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:35.775645 [slave.cpp:3002] Handling status update TASK_KILLED (UUID:
eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 f
rom @0.0.0.0:0
13:33:35.778105 [cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1) for
container ead42e63-ac92-4ad0-a99c-4af9c3fa5e31
13:33:35.778488 [disk.cpp:169] Updating the disk resources for container
ead42e63-ac92-4ad0-a99c-4af9c3fa5e31 to cpus(*):0.1; mem(*):32
13:33:35.780349 [mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 32MB for
container ead42e63-ac92-4ad0-a99c-4af9c3fa5e3
1
13:33:35.782573 [status_update_manager.cpp:320] Received status update
TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a8
1b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-9173
9fa4394c-0000
13:33:35.783860 [status_update_manager.cpp:824] Checkpointing UPDATE for
status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:35.788767 [slave.cpp:3400] Forwarding the update TASK_KILLED (UUID:
eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 to [email protected]:5050
13:33:35.917932 [status_update_manager.cpp:392] Received status update
acknowledgement (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:35.918143 [status_update_manager.cpp:824] Checkpointing ACK for status
update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
...
13:33:39.031054 [slave.cpp:2643] Got registration for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 from executor(1)@127.0.0.1:60083
{code}
was:
I'm running Mesos 0.28.0. Mesos slave(1)/state endpoint returns some completed
executors not in frameworks.completed_executors but in frameworks.executors.
{code:JavaScript:title=slave(1)/state}
{
"attributes": {...},
"completed_frameworks": [],
"flags": {...},
"frameworks": [
{
"checkpoint": true,
"completed_executors": [...],
"executors": [
{
"queued_tasks": [],
"tasks": [],
"completed_tasks": [
{
"discovery": {...},
"executor_id": "",
"framework_id":
"f65b163c-0faf-441f-ac14-91739fa4394c-0000",
"id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
"labels": [...],
"name": "service",
"resources": {...},
"slave_id":
"ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13",
"state": "TASK_KILLED",
"statuses": []
}
],
"container": "ead42e63-ac92-4ad0-a99c-4af9c3fa5e31",
"directory": "...",
"id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
"name": "Command Executor (Task:
service.a3b609b8-27ec-11e6-8044-02c89eb9127e) (Command: sh -c 'cd
service...')",
"resources": {...},
"source": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e"
},
...
],
}
],
"git_sha": "961edbd82e691a619a4c171a7aadc9c32957fa73",
"git_tag": "0.28.0",
"version": "0.28.0",
...
}
{code}
{code:title="var/log/mesos/mesos-slave.INFO"}
13:33:19.479182 [slave.cpp:1361] Got assigned task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:19.482566 [slave.cpp:1480] Launching task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:19.483921 [paths.cpp:528] Trying to chown
'/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
to user 'mesosuser'
13:33:19.504173 [slave.cpp:5367] Launching executor
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 with resources cpus(*):0.1; mem(*):32
in work directory
'/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
13:33:19.505537 [containerizer.cpp:666] Starting container
'ead42e63-ac92-4ad0-a99c-4af9c3fa5e31' for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
'f65b163c-0faf-441f-ac14-91739fa4394c-0000'
13:33:19.505734 [slave.cpp:1698] Queuing task
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
...
13:33:19.977483 [containerizer.cpp:1118] Checkpointing executor's forked pid
25576 to
'/tmp/mesos/meta/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31/pids/forked.pid'
13:33:35.775195 [slave.cpp:1891] Asked to kill task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:35.775645 [slave.cpp:3002] Handling status update TASK_KILLED (UUID:
eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 f
rom @0.0.0.0:0
13:33:35.778105 [cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1) for
container ead42e63-ac92-4ad0-a99c-4af9c3fa5e31
13:33:35.778488 [disk.cpp:169] Updating the disk resources for container
ead42e63-ac92-4ad0-a99c-4af9c3fa5e31 to cpus(*):0.1; mem(*):32
13:33:35.780349 [mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 32MB for
container ead42e63-ac92-4ad0-a99c-4af9c3fa5e3
1
13:33:35.782573 [status_update_manager.cpp:320] Received status update
TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a8
1b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-9173
9fa4394c-0000
13:33:35.783860 [status_update_manager.cpp:824] Checkpointing UPDATE for
status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:35.788767 [slave.cpp:3400] Forwarding the update TASK_KILLED (UUID:
eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 to [email protected]:5050
13:33:35.917932 [status_update_manager.cpp:392] Received status update
acknowledgement (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
13:33:35.918143 [status_update_manager.cpp:824] Checkpointing ACK for status
update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000
...
13:33:39.031054 [slave.cpp:2643] Got registration for executor
'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
f65b163c-0faf-441f-ac14-91739fa4394c-0000 from executor(1)@127.0.0.1:60083
{code}
> Completed executors presented as alive
> --------------------------------------
>
> Key: MESOS-5536
> URL: https://issues.apache.org/jira/browse/MESOS-5536
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 0.28.0
> Environment: Ubuntu 14.04.3 LTS
> Reporter: Tomasz Janiszewski
>
> I'm running Mesos 0.28.0. Mesos {{slave(1)/state}} endpoint returns some
> completed executors not in frameworks.completed_executors but in
> frameworks.executors. Alsa this executor presents in {{monitor/statistics}}
> {code:JavaScript:title=slave(1)/state}
> {
> "attributes": {...},
> "completed_frameworks": [],
> "flags": {...},
> "frameworks": [
> {
> "checkpoint": true,
> "completed_executors": [...],
> "executors": [
> {
> "queued_tasks": [],
> "tasks": [],
> "completed_tasks": [
> {
> "discovery": {...},
> "executor_id": "",
> "framework_id":
> "f65b163c-0faf-441f-ac14-91739fa4394c-0000",
> "id":
> "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
> "labels": [...],
> "name": "service",
> "resources": {...},
> "slave_id":
> "ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13",
> "state": "TASK_KILLED",
> "statuses": []
> }
> ],
> "container": "ead42e63-ac92-4ad0-a99c-4af9c3fa5e31",
> "directory": "...",
> "id": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e",
> "name": "Command Executor (Task:
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e) (Command: sh -c 'cd
> service...')",
> "resources": {...},
> "source": "service.a3b609b8-27ec-11e6-8044-02c89eb9127e"
>
> },
> ...
> ],
> }
> ],
> "git_sha": "961edbd82e691a619a4c171a7aadc9c32957fa73",
> "git_tag": "0.28.0",
> "version": "0.28.0",
> ...
> }
> {code}
> {code:title="var/log/mesos/mesos-slave.INFO"}
> 13:33:19.479182 [slave.cpp:1361] Got assigned task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> 13:33:19.482566 [slave.cpp:1480] Launching task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e for framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> 13:33:19.483921 [paths.cpp:528] Trying to chown
> '/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
> to user 'mesosuser'
> 13:33:19.504173 [slave.cpp:5367] Launching executor
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/tmp/mesos/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31'
> 13:33:19.505537 [containerizer.cpp:666] Starting container
> 'ead42e63-ac92-4ad0-a99c-4af9c3fa5e31' for executor
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
> 'f65b163c-0faf-441f-ac14-91739fa4394c-0000'
> 13:33:19.505734 [slave.cpp:1698] Queuing task
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' for executor
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> ...
> 13:33:19.977483 [containerizer.cpp:1118] Checkpointing executor's forked pid
> 25576 to
> '/tmp/mesos/meta/slaves/ef232fd9-5114-4d8f-adc3-1669c1e6fdc5-S13/frameworks/f65b163c-0faf-441f-ac14-91739fa4394c-0000/executors/service.a3b609b8-27ec-11e6-8044-02c89eb9127e/runs/ead42e63-ac92-4ad0-a99c-4af9c3fa5e31/pids/forked.pid'
> 13:33:35.775195 [slave.cpp:1891] Asked to kill task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> 13:33:35.775645 [slave.cpp:3002] Handling status update TASK_KILLED (UUID:
> eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 f
> rom @0.0.0.0:0
> 13:33:35.778105 [cpushare.cpp:389] Updated 'cpu.shares' to 102 (cpus 0.1)
> for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e31
> 13:33:35.778488 [disk.cpp:169] Updating the disk resources for container
> ead42e63-ac92-4ad0-a99c-4af9c3fa5e31 to cpus(*):0.1; mem(*):32
> 13:33:35.780349 [mem.cpp:353] Updated 'memory.soft_limit_in_bytes' to 32MB
> for container ead42e63-ac92-4ad0-a99c-4af9c3fa5e3
> 1
> 13:33:35.782573 [status_update_manager.cpp:320] Received status update
> TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a8
> 1b) for task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-9173
> 9fa4394c-0000
> 13:33:35.783860 [status_update_manager.cpp:824] Checkpointing UPDATE for
> status update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for
> task service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> 13:33:35.788767 [slave.cpp:3400] Forwarding the update TASK_KILLED (UUID:
> eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 to [email protected]:5050
> 13:33:35.917932 [status_update_manager.cpp:392] Received status update
> acknowledgement (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> 13:33:35.918143 [status_update_manager.cpp:824] Checkpointing ACK for status
> update TASK_KILLED (UUID: eba64915-7df2-483d-8982-a9a46a48a81b) for task
> service.a3b609b8-27ec-11e6-8044-02c89eb9127e of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000
> ...
> 13:33:39.031054 [slave.cpp:2643] Got registration for executor
> 'service.a3b609b8-27ec-11e6-8044-02c89eb9127e' of framework
> f65b163c-0faf-441f-ac14-91739fa4394c-0000 from executor(1)@127.0.0.1:60083
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
