When Mesos is asked to a launch a task (with either a custom Executor or the
built in CommandExecutor) it will first spawn the executor which _has_ to be a
system process, launched via command. This process will be launched inside of a
Docker container when using the previously mentioned containerizers.
Once the Executor registers with the slave, the slave will send it a number of
launchTask calls based on the number of tasks queued up for that executor. The
Executor can then do as it pleases with those tasks, whether it's just a
sleep(1) or to spawn a subprocess and do some other work. Given it is possible
for the framework to specify resources for both tasks and executors, and the
only thing which _has_ to be a system process is the executor, the mesos slave
will limit the resources of the executor process to the sum of
(TaskInfo.Executor.Resources + TaskInfo.Resources).
Mesos also has the ability to launch new tasks on an already running executor,
so it's important that mesos is able to dynamically scale the resource limits
up and down over time. Designing a framework around this idea can lead to some
complex and powerful workflows which would be a lot more complex to build
without Mesos.
Just for an example... Spark.
1) User launches a job on spark to map over some data
2) Spark launches a first wave of tasks based on the offers it received (let's
say T1 and T2)
3) Mesos launches executors for those tasks (let's say E1 and E2) on different
slaves
4) Spark launches another wave of tasks based on offers, and tells mesos to use
the same executor (E1 and E2)
5) Mesos will simply call launchTasks(T{3,4}) on the two already running
executors
At point (3) mesos is going to launch a Docker container and execute your
executor. However at (5) the executor is already running so the tasks will be
handed to the already running executor.
Mesos will guarantee you (i'm 99% sure) that the resources for your container
have been updated to reflect the limits set on the tasks before handing the
tasks to you.
I hope that makes some sense!
--
Tom Arnfeld
Developer // DueDil
On Wed, Dec 3, 2014 at 10:54 AM, Diptanu Choudhury <dipta...@gmail.com>
wrote:
> Thanks for the explanation Tom, yeah I just figured that out by reading
> your code! You're touching the memory.soft_limit_in_bytes and
> memory.limit_in_bytes directly.
> Still curios to understand in which situations Mesos Slave would call the
> external containerizer to update the resource limits of a container? My
> understanding was that once resource allocation happens for a task,
> resources are not taken away until the task exits[fails, crashes or
> finishes] or Mesos asks the slave to kill the task.
> On Wed, Dec 3, 2014 at 2:47 AM, Tom Arnfeld <t...@duedil.com> wrote:
>> Hi Diptanu,
>>
>> That's correct, the ECP has the responsibility of updating the resource
>> for a container, and it will do as new tasks are launched and killed for an
>> executor. Since docker doesn't support this, our containerizer (Deimos does
>> the same) goes behind docker to the cgroup for the container and updates
>> the resources in a very similar way to the mesos-slave. I believe this is
>> also what the built in Docker containerizer will do.
>>
>>
>> https://github.com/duedil-ltd/mesos-docker-containerizer/blob/master/containerizer/commands/update.py#L35
>>
>> Tom.
>>
>> --
>>
>> Tom Arnfeld
>> Developer // DueDil
>>
>>
>> On Wed, Dec 3, 2014 at 10:45 AM, Diptanu Choudhury <dipta...@gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I had a quick question about the external containerizer. I see that once
>>> the Task is launched, the ECP can receive the update calls, and the
>>> protobuf message passed to ECP with the update call is
>>> containerizer::Update.
>>>
>>> This protobuf has a Resources [list] field so does that mean Mesos might
>>> ask a running task to re-adjust the enforced resource limits?
>>>
>>> How would this work if the ECP was launching docker containers because
>>> Docker doesn't allow changing the resource limits once the container has
>>> been started?
>>>
>>> I am wondering how does Deimos and mesos-docker-containerizer handle this.
>>>
>>> --
>>> Thanks,
>>> Diptanu Choudhury
>>> Web - www.linkedin.com/in/diptanu
>>> Twitter - @diptanu <http://twitter.com/diptanu>
>>>
>>
>>
> --
> Thanks,
> Diptanu Choudhury
> Web - www.linkedin.com/in/diptanu
> Twitter - @diptanu <http://twitter.com/diptanu>
