Hi Tim, I'm using Fine grain mode? Is it also intended behaviour in that mode?
Jongyoul On Fri, Jan 2, 2015 at 5:33 PM, Timothy Chen <[email protected]> wrote: > Hi Jongyul, > > Are you using Coarse grain or Fine grain mode? > > In coarse grain mode all tasks are sharing the same spark and mesos > executor, so the logs are all forwarded to the same stderr/stdout > files. > > Tim > > On Fri, Jan 2, 2015 at 12:25 AM, Jongyoul Lee <[email protected]> wrote: > > Hi Tim, > > > > I've found a case that two different task runs under same containerId > > simultaneously when I've run a heavy - that means that job use all > > resources - spark job. Spark mesos scheduler creates ExecutorInfo with > > slaveId when it launches task. I saw two different tasks write logs as > same > > file (in same directory including containerId) Is It possible? Sorry for > an > > ambiguous question. > > > > JL > > > > On Fri, Jan 2, 2015 at 5:05 PM, Timothy Chen <[email protected]> wrote: > > > >> Hi Jongyoul, > >> > >> Strictly speaking, Mesos containerizer creates a container, and > >> launches the executor inside of that container which launches the > >> task. > >> > >> The code path you're looking at is in the slave side (slave.cpp), > >> which calls containerizer->launch when executor is not found. > >> > >> Either way your observation is correct, that launchExecutor in the > >> slave is only called when a executor is not found in the framework > >> object internally in the slave object. > >> > >> Tim > >> > >> > >> > >> On Fri, Jan 2, 2015 at 12:00 AM, Jongyoul Lee <[email protected]> > wrote: > >> > Thanks, Nishant and Tim, > >> > > >> > I'm using mesos_containerizer for running spark tasks. In case of > >> > mesos_containerizer - or anyone else -, Could you please tell me when > >> > executor creates a new container or give me more hint to find that? > I've > >> > found code that framework->launchExecutor is called when > >> > framework->getExecutor returns null. > >> > > >> > JL > >> > > >> > On Fri, Jan 2, 2015 at 4:17 AM, Timothy Chen <[email protected]> > wrote: > >> > > >> >> Hi Jongyoul, > >> >> > >> >> It depends on what you refer to as a "container". > >> >> > >> >> In Mesos actually there are several concepts in play, > >> >> tasks that is either a custom executor or a command, executors that > >> >> are responsible for launching tasks, and containerizers that are > >> >> responsible for creating the containers to launch executors and/or > >> >> tasks. > >> >> > >> >> So if you are referring to an actual container that in mesos is > either > >> >> cgroups or docker, then depending on the implementation of the > >> >> particular containerizer (Mesos or Docker or External, etc), a > >> >> container can have multiple tasks or executors although currently all > >> >> of them launches one container per executor. Executor itself can > >> >> potentially launch multiple tasks that are running simultaneously as > >> >> long as the executor is registered and referred to the same id by > >> >> multiple tasks. > >> >> > >> >> In the slave object itself, it only tracks frameworks and executors, > >> >> and when it doens't find it registered in memory it will ask the > >> >> containerizer to launch it. > >> >> > >> >> Tim > >> >> > >> >> > >> >> > >> >> > >> >> On Thu, Jan 1, 2015 at 11:01 AM, Nishant Suneja < > >> [email protected]> > >> >> wrote: > >> >> > Hi Jongyoul > >> >> > > >> >> > Take a look at the slave/state.hpp file, and go through the > hierarchy > >> of > >> >> > the state information starting from slave to task. > >> >> > You will see that an executor state has a map of run states, which > is > >> >> > indexed by containerId. > >> >> > It also has "latest" field to denote the current containerId which > its > >> >> > using. > >> >> > > >> >> > So, to answer your question, at any point of time, an executor > should > >> >> have > >> >> > just one active container (which means one process on the system), > >> >> > and that process's working directory would be identified by the > >> "latest" > >> >> > containerId field in the ExecutorState. > >> >> > But, it can have multiple directories from the previous runs, each > >> >> > identified by a unique containerId. > >> >> > These directories would eventually be garbage collected. > >> >> > > >> >> > Nishant > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > On Tue, Dec 30, 2014 at 6:12 PM, Jongyoul Lee <[email protected]> > >> >> wrote: > >> >> > > >> >> >> Hi, dev > >> >> >> > >> >> >> I'm curious about the relationship between executor and container. > >> Can > >> >> any > >> >> >> executor have multiple container? or have just one? In codes of > >> >> slave.cpp, > >> >> >> launchExecutor only can create container and is called only when > the > >> >> return > >> >> >> value of getExecutor is null. Thus I think each executor can have > one > >> >> >> container only. Is it correct? If I'm wrong, please tell me > correct > >> >> >> procedure. > >> >> >> > >> >> >> Thanks in advance, > >> >> >> JL > >> >> >> > >> >> >> -- > >> >> >> 이종열, Jongyoul Lee, 李宗烈 > >> >> >> http://madeng.net > >> >> >> > >> >> > >> > > >> > > >> > > >> > -- > >> > 이종열, Jongyoul Lee, 李宗烈 > >> > http://madeng.net > >> > > > > > > > > -- > > 이종열, Jongyoul Lee, 李宗烈 > > http://madeng.net > -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net
