Charles Natali created MESOS-10119:
--------------------------------------
Summary: failure to destroy container can cause the agent to
"leak" a GPU
Key: MESOS-10119
URL: https://issues.apache.org/jira/browse/MESOS-10119
Project: Mesos
Issue Type: Task
Components: agent, containerization
Reporter: Charles Natali
At work we hit the following problem:
# cgroup for a task using the GPU isolation failed to be destroy on OOM
# the agent continued advertising the GPU as available
# all subsequent attempts to start tasks using a GPU fails with "Requested 1
gpus but only 0 available"
Problem 1 looks like https://issues.apache.org/jira/browse/MESOS-9950) so can
be tackled separately, however the fact that the agent basically leaks the GPU
is pretty bad, because it basically turns into /dev/null, failing all
subsequent tasks requesting a gpu
See the logs:
{noformat}
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874277 2138
memory.cpp:665] Failed to read 'memory.limit_in_bytes': No such file or
directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874305 2138
memory.cpp:674] Failed to read 'memory.max_usage_in_bytes': No such file or
directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874315 2138
memory.cpp:686] Failed to read 'memory.stat': No such file or directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874701 2136
memory.cpp:665] Failed to read 'memory.limit_in_bytes': No such file or
directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874734 2136
memory.cpp:674] Failed to read 'memory.max_usage_in_bytes': No such file or
directory
Apr 17 17:00:03 engpuc006 mesos-slave[2068]: E0417 17:00:03.874747 2136
memory.cpp:686] Failed to read 'memory.stat': No such file or directory
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: E0417 17:00:05.062358 2152
slave.cpp:6994] Termination of executor
'task_0:067b0963-134f-a917-4503-89b6a2a630ac' of framework
c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed: Failed to clean up an
isolator when destroying container: Failed to destroy cgroups: Failed to get
nested cgroups: Failed to determine canonical path of
'/sys/fs/cgroup/memory/mesos/8ef00748-b640-4620-97dc-f719e9775e88': No such
file or directory
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.063295 2150
containerizer.cpp:2567] Skipping status for container
8ef00748-b640-4620-97dc-f719e9775e88 because: Container does not exist
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.063429 2137
containerizer.cpp:2428] Ignoring update for currently being destroyed container
8ef00748-b640-4620-97dc-f719e9775e88
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: E0417 17:00:05.079169 2150
slave.cpp:6994] Termination of executor
'task_1:a00165a1-123a-db09-6b1a-b6c4054b0acd' of framework
c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed: Failed to kill all processes
in the container: Failed to remove cgroup
'mesos/5c1418f0-1d4d-47cd-a188-0f4b87e394f2': Failed to remove cgroup
'/sys/fs/cgroup/freezer/mesos/5c1418f0-1d4d-47cd-a188-0f4b87e394f2': Device or
resource busy
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.079537 2140
containerizer.cpp:2567] Skipping status for container
5c1418f0-1d4d-47cd-a188-0f4b87e394f2 because: Container does not exist
Apr 17 17:00:05 engpuc006 mesos-slave[2068]: W0417 17:00:05.079670 2136
containerizer.cpp:2428] Ignoring update for currently being destroyed container
5c1418f0-1d4d-47cd-a188-0f4b87e394f2
Apr 17 17:00:07 engpuc006 mesos-slave[2068]: E0417 17:00:07.956969 2136
slave.cpp:6889] Container '87253521-8d39-47ea-b4d1-febe527d230c' for executor
'task_2:8b129d24-70d2-2cab-b2df-c73911954ec3' of framework
c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed to start: Requested 1 gpus but
only 0 available
Apr 17 17:00:07 engpuc006 mesos-slave[2068]: E0417 17:00:07.957670 2149
memory.cpp:637] Listening on OOM events failed for container
87253521-8d39-47ea-b4d1-febe527d230c: Event listener is terminating
Apr 17 17:00:07 engpuc006 mesos-slave[2068]: W0417 17:00:07.966552 2150
containerizer.cpp:2421] Ignoring update for unknown container
87253521-8d39-47ea-b4d1-febe527d230c
Apr 17 17:00:08 engpuc006 mesos-slave[2068]: W0417 17:00:08.109067 2154
process.cpp:1480] Failed to link to '172.16.22.201:34059', connect: Failed
connect: connection closed
Apr 17 17:00:10 engpuc006 mesos-slave[2068]: E0417 17:00:10.310817 2141
slave.cpp:6889] Container '257b45f1-8582-4cb5-8138-454e9697bfe4' for executor
'task_3:6bdd99ca-7a2b-f19c-bbb3-d9478fe8f81e' of framework
c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed to start: Requested 1 gpus but
only 0 available
Apr 17 17:00:10 engpuc006 mesos-slave[2068]: E0417 17:00:10.311614 2141
memory.cpp:637] Listening on OOM events failed for container
257b45f1-8582-4cb5-8138-454e9697bfe4: Event listener is terminating
Apr 17 17:00:10 engpuc006 mesos-slave[2068]: W0417 17:00:10.316352 2149
containerizer.cpp:2421] Ignoring update for unknown container
257b45f1-8582-4cb5-8138-454e9697bfe4
Apr 17 17:00:12 engpuc006 mesos-slave[2068]: E0417 17:00:12.675910 2146
slave.cpp:6889] Container '47f61e7b-8022-4731-93fe-36a356920a4e' for executor
'task_2:9de8db83-39fa-b24a-b5b6-2ef6f1e1713c' of framework
c0c4ce82-5cff-4116-aacb-c3fd6a93d61b-0000 failed to start: Requested 1 gpus but
only 0 available
Apr 17 17:00:12 engpuc006 mesos-slave[2068]: E0417 17:00:12.676764 2139
memory.cpp:637] Listening on OOM events failed for container
47f61e7b-8022-4731-93fe-36a356920a4e: Event listener is terminating
Apr 17 17:00:12 engpuc006 mesos-slave[2068]: W0417 17:00:12.681100 2150
containerizer.cpp:2421] Ignoring update for unknown container
47f61e7b-8022-4731-93fe-36a356920a4e
{noformat}
I'll see if I can reproduce and debug the cgroup destruction problem and report
my findings to https://issues.apache.org/jira/browse/MESOS-9950), however I
think that the bigger issue is the fact that the agent leaks the GPU.
I'm wondering which how the following makes sense:
# Consider the failure of destruction a container a fatal error: this turns
this byzantine fault into a clean crash failure, which forces the operator to
detect it and deal with it.
# Fix the leak by at least not advertising the GPU anymore.
[~bmahler] [~abudnik]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
