[ https://issues.apache.org/jira/browse/MESOS-8209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kone reassigned MESOS-8209: --------------------------------- Assignee: Vinod Kone > mesos master should revoke offers when executor state changes > ------------------------------------------------------------- > > Key: MESOS-8209 > URL: https://issues.apache.org/jira/browse/MESOS-8209 > Project: Mesos > Issue Type: Bug > Reporter: Jack Crawford > Assignee: Vinod Kone > > Currently, the mesos master does not revoke offers when the number of > executors on an agent decreases. This is a problem under certain conditions, > such when running a workflow that starts lots of small tasks on agents, with > a one executor per task model, a master that does not revoke resources after > a set amount of time, and a scheduler that does not reject resources. > The problem is that when running a mono-scheduler framework (which you might > want to do to easily enforce authentication requirements, have a full view of > all scheduled tasks, etc), in order to respond instantly when new tasks come > in I have the scheduler simply hang on to all resource offers it receives, > and the master is set to never revoke offers. This way the scheduler always > has a pool of resources to quickly service new requests as they come in. > However, if you start tasks fast enough, the agents can fill up with > executors, making it appear as there are no resources available for the > scheduler to use. Ive seen this on r4.4xlarge machines on aws with executors > that consume 0.1 cpus, 32mb mem where the entire machine will be appear to be > filled with executors according to the master resource offers. The executors > are exiting (just after the task finishes), but the resources are not > reclaimed because the master does not revoke the outstanding resource offers > to reflect the change. > You can replicate this pretty easily if you schedule tasks that finish > instantly with a 1-1 executor to task ratio. I find that if I schedule ~1000 > tasks this way on a single r4.4xlarge machine, usually 600-700 will finish > before all the resource offers to the scheduler fill up and the agent appears > to be "full" of executors. > Changing the scheduler/master to periodically reject/revoke resources fixes > the problem. > My suggestion is for the master to revoke and reissue resource offers when > the executor count changes on an agent. -- This message was sent by Atlassian JIRA (v6.4.14#64029)