Hi Jongyul, Are you using Coarse grain or Fine grain mode?
In coarse grain mode all tasks are sharing the same spark and mesos executor, so the logs are all forwarded to the same stderr/stdout files. Tim On Fri, Jan 2, 2015 at 12:25 AM, Jongyoul Lee <[email protected]> wrote: > Hi Tim, > > I've found a case that two different task runs under same containerId > simultaneously when I've run a heavy - that means that job use all > resources - spark job. Spark mesos scheduler creates ExecutorInfo with > slaveId when it launches task. I saw two different tasks write logs as same > file (in same directory including containerId) Is It possible? Sorry for an > ambiguous question. > > JL > > On Fri, Jan 2, 2015 at 5:05 PM, Timothy Chen <[email protected]> wrote: > >> Hi Jongyoul, >> >> Strictly speaking, Mesos containerizer creates a container, and >> launches the executor inside of that container which launches the >> task. >> >> The code path you're looking at is in the slave side (slave.cpp), >> which calls containerizer->launch when executor is not found. >> >> Either way your observation is correct, that launchExecutor in the >> slave is only called when a executor is not found in the framework >> object internally in the slave object. >> >> Tim >> >> >> >> On Fri, Jan 2, 2015 at 12:00 AM, Jongyoul Lee <[email protected]> wrote: >> > Thanks, Nishant and Tim, >> > >> > I'm using mesos_containerizer for running spark tasks. In case of >> > mesos_containerizer - or anyone else -, Could you please tell me when >> > executor creates a new container or give me more hint to find that? I've >> > found code that framework->launchExecutor is called when >> > framework->getExecutor returns null. >> > >> > JL >> > >> > On Fri, Jan 2, 2015 at 4:17 AM, Timothy Chen <[email protected]> wrote: >> > >> >> Hi Jongyoul, >> >> >> >> It depends on what you refer to as a "container". >> >> >> >> In Mesos actually there are several concepts in play, >> >> tasks that is either a custom executor or a command, executors that >> >> are responsible for launching tasks, and containerizers that are >> >> responsible for creating the containers to launch executors and/or >> >> tasks. >> >> >> >> So if you are referring to an actual container that in mesos is either >> >> cgroups or docker, then depending on the implementation of the >> >> particular containerizer (Mesos or Docker or External, etc), a >> >> container can have multiple tasks or executors although currently all >> >> of them launches one container per executor. Executor itself can >> >> potentially launch multiple tasks that are running simultaneously as >> >> long as the executor is registered and referred to the same id by >> >> multiple tasks. >> >> >> >> In the slave object itself, it only tracks frameworks and executors, >> >> and when it doens't find it registered in memory it will ask the >> >> containerizer to launch it. >> >> >> >> Tim >> >> >> >> >> >> >> >> >> >> On Thu, Jan 1, 2015 at 11:01 AM, Nishant Suneja < >> [email protected]> >> >> wrote: >> >> > Hi Jongyoul >> >> > >> >> > Take a look at the slave/state.hpp file, and go through the hierarchy >> of >> >> > the state information starting from slave to task. >> >> > You will see that an executor state has a map of run states, which is >> >> > indexed by containerId. >> >> > It also has "latest" field to denote the current containerId which its >> >> > using. >> >> > >> >> > So, to answer your question, at any point of time, an executor should >> >> have >> >> > just one active container (which means one process on the system), >> >> > and that process's working directory would be identified by the >> "latest" >> >> > containerId field in the ExecutorState. >> >> > But, it can have multiple directories from the previous runs, each >> >> > identified by a unique containerId. >> >> > These directories would eventually be garbage collected. >> >> > >> >> > Nishant >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > On Tue, Dec 30, 2014 at 6:12 PM, Jongyoul Lee <[email protected]> >> >> wrote: >> >> > >> >> >> Hi, dev >> >> >> >> >> >> I'm curious about the relationship between executor and container. >> Can >> >> any >> >> >> executor have multiple container? or have just one? In codes of >> >> slave.cpp, >> >> >> launchExecutor only can create container and is called only when the >> >> return >> >> >> value of getExecutor is null. Thus I think each executor can have one >> >> >> container only. Is it correct? If I'm wrong, please tell me correct >> >> >> procedure. >> >> >> >> >> >> Thanks in advance, >> >> >> JL >> >> >> >> >> >> -- >> >> >> 이종열, Jongyoul Lee, 李宗烈 >> >> >> http://madeng.net >> >> >> >> >> >> > >> > >> > >> > -- >> > 이종열, Jongyoul Lee, 李宗烈 >> > http://madeng.net >> > > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net
