Hi Tim, I've found a case that two different task runs under same containerId simultaneously when I've run a heavy - that means that job use all resources - spark job. Spark mesos scheduler creates ExecutorInfo with slaveId when it launches task. I saw two different tasks write logs as same file (in same directory including containerId) Is It possible? Sorry for an ambiguous question.
JL On Fri, Jan 2, 2015 at 5:05 PM, Timothy Chen <[email protected]> wrote: > Hi Jongyoul, > > Strictly speaking, Mesos containerizer creates a container, and > launches the executor inside of that container which launches the > task. > > The code path you're looking at is in the slave side (slave.cpp), > which calls containerizer->launch when executor is not found. > > Either way your observation is correct, that launchExecutor in the > slave is only called when a executor is not found in the framework > object internally in the slave object. > > Tim > > > > On Fri, Jan 2, 2015 at 12:00 AM, Jongyoul Lee <[email protected]> wrote: > > Thanks, Nishant and Tim, > > > > I'm using mesos_containerizer for running spark tasks. In case of > > mesos_containerizer - or anyone else -, Could you please tell me when > > executor creates a new container or give me more hint to find that? I've > > found code that framework->launchExecutor is called when > > framework->getExecutor returns null. > > > > JL > > > > On Fri, Jan 2, 2015 at 4:17 AM, Timothy Chen <[email protected]> wrote: > > > >> Hi Jongyoul, > >> > >> It depends on what you refer to as a "container". > >> > >> In Mesos actually there are several concepts in play, > >> tasks that is either a custom executor or a command, executors that > >> are responsible for launching tasks, and containerizers that are > >> responsible for creating the containers to launch executors and/or > >> tasks. > >> > >> So if you are referring to an actual container that in mesos is either > >> cgroups or docker, then depending on the implementation of the > >> particular containerizer (Mesos or Docker or External, etc), a > >> container can have multiple tasks or executors although currently all > >> of them launches one container per executor. Executor itself can > >> potentially launch multiple tasks that are running simultaneously as > >> long as the executor is registered and referred to the same id by > >> multiple tasks. > >> > >> In the slave object itself, it only tracks frameworks and executors, > >> and when it doens't find it registered in memory it will ask the > >> containerizer to launch it. > >> > >> Tim > >> > >> > >> > >> > >> On Thu, Jan 1, 2015 at 11:01 AM, Nishant Suneja < > [email protected]> > >> wrote: > >> > Hi Jongyoul > >> > > >> > Take a look at the slave/state.hpp file, and go through the hierarchy > of > >> > the state information starting from slave to task. > >> > You will see that an executor state has a map of run states, which is > >> > indexed by containerId. > >> > It also has "latest" field to denote the current containerId which its > >> > using. > >> > > >> > So, to answer your question, at any point of time, an executor should > >> have > >> > just one active container (which means one process on the system), > >> > and that process's working directory would be identified by the > "latest" > >> > containerId field in the ExecutorState. > >> > But, it can have multiple directories from the previous runs, each > >> > identified by a unique containerId. > >> > These directories would eventually be garbage collected. > >> > > >> > Nishant > >> > > >> > > >> > > >> > > >> > > >> > On Tue, Dec 30, 2014 at 6:12 PM, Jongyoul Lee <[email protected]> > >> wrote: > >> > > >> >> Hi, dev > >> >> > >> >> I'm curious about the relationship between executor and container. > Can > >> any > >> >> executor have multiple container? or have just one? In codes of > >> slave.cpp, > >> >> launchExecutor only can create container and is called only when the > >> return > >> >> value of getExecutor is null. Thus I think each executor can have one > >> >> container only. Is it correct? If I'm wrong, please tell me correct > >> >> procedure. > >> >> > >> >> Thanks in advance, > >> >> JL > >> >> > >> >> -- > >> >> 이종열, Jongyoul Lee, 李宗烈 > >> >> http://madeng.net > >> >> > >> > > > > > > > > -- > > 이종열, Jongyoul Lee, 李宗烈 > > http://madeng.net > -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net
