I filed one: https://issues.apache.org/jira/browse/MESOS-8167
It's a pretty significant effort, and hasn't been requested a lot, so it's
unlikely to be worked on for some time.
On Tue, Oct 31, 2017 at 8:18 PM, Mohit Jaggi wrote:
> :-)
> Is there a Jira ticket to track this? Any idea when this will be worked on?
>
> On Tue, Oct 31, 2017 at 5:22 PM, Benjamin Mahler
> wrote:
>
>> The question was posed merely to point out that there is no notion of the
>> executor "running away" currently, due to the answer I provided: there
>> isn't a complete lifecycle API for the executor. (This includes
>> healthiness, state updates, reconciliation, ability for scheduler to shut
>> it down, etc).
>>
>> On Tue, Oct 31, 2017 at 4:27 PM, Mohit Jaggi
>> wrote:
>>
>>> Good question.
>>> - I don't know what the interaction between mesos agent and executor is.
>>> Is there a health check?
>>> - There is a reconciliation between Mesos and Frameworks: will Mesos
>>> include the "orphan" executor in the list there, so framework can find
>>> runaways and kill them(using Mesos provided API)?
>>>
>>> On Tue, Oct 31, 2017 at 3:49 PM, Benjamin Mahler
>>> wrote:
>>>
What defines a runaway executor?
Mesos does not know that this particular executor should self-terminate
within some reasonable time after its task terminates. In this case the
framework (Aurora) knows this expected behavior of Thermos and can clean up
ones that get stuck after the task terminates. However, we currently don't
provide a great executor lifecycle API to enable schedulers to do this
(it's long overdue).
On Tue, Oct 31, 2017 at 2:47 PM, Mohit Jaggi
wrote:
> I was asking if this can happen automatically.
>
> On Tue, Oct 31, 2017 at 2:41 PM, Benjamin Mahler
> wrote:
>
>> You can kill it manually by SIGKILLing the executor process.
>> Using the agent API, you can launch a nested container session and
>> kill the executor. +jie,gilbert, is there a CLI command for 'exec'ing
>> into
>> the container?
>>
>> On Tue, Oct 31, 2017 at 12:47 PM, Mohit Jaggi
>> wrote:
>>
>>> Yes. There is a fix available now in Aurora/Thermos to try and exit
>>> in such scenarios. But I am curious to know if Mesos agent has the
>>> functionality to reap runaway executors.
>>>
>>> On Tue, Oct 31, 2017 at 12:08 PM, Benjamin Mahler <
>>> bmah...@apache.org> wrote:
>>>
Is my understanding correct that the Thermos transitions the task
to TASK_FAILED, but Thermos gets stuck and can't terminate itself? The
typical workflow for thermos, as a 1:1 task:executor approach, is that
the
executor terminates itself after the task is terminal.
The full logs of the agent during this window would help, it looks
like an agent termination is involved here as well?
On Fri, Oct 27, 2017 at 3:09 PM, Mohit Jaggi
wrote:
> Here are some relevant logs. Aurora scheduler logs shows the task
> going from:
> INIT
> ->PENDING
> ->ASSIGNED
> ->STARTING
> ->RUNNING for a long time
> ->FAILED due to health check error, OSError: Resource temporarily
> unavailable (I think this is referring to running out of PID space,
> see
> thermos logs below)
>
>
> --- mesos agent ---
>
> I1005 22:56:47.902153 127818 fetcher.cpp:285] Fetching directly into
> the sandbox directory
> I1005 22:56:47.902170 127818 fetcher.cpp:222] Fetching URI
> '/usr/bin/X'
> I1005 22:56:47.913270 127818 fetcher.cpp:207] Copied resource
> '/usr/bin/x' to
> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
> I1005 22:56:47.913331 127818 fetcher.cpp:582] Fetched '/usr/bin/xxx'
> to
> '/var/lib/mesos/slaves/b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540/frameworks/20160112-010512-421372426-5050-73504-/executors/thermos-xxx-2-caa0744d-fffd-446e-9f97-05bd84a32b54/runs/bb904e1d-4c32-4d7a-b1b6-9b3f78ddfe95/xxx'
> WARNING: Your kernel does not support swap limit capabilities, memory
> limited without swap.
> twitter.common.app debug: Initializing: twitter.common.log (Logging
> subsystem.)
> Writing log files to disk in /mnt/mesos/sandbox
> I1005 22:58:15.677225 7 exec.cpp:162] Version: 1.1.0
> I1005 22:58:15.68086714 exec.cpp:237] Executor registered on
> agent b4fff262-c925-4edf-a2ef-2a5bbe89c42b-S1540
> Writing log files to disk in /mnt/mesos/sandbox
> I1006 01:13:52.95055239 exec.cpp:487] Agent exited, but framework
> has
