Hi Tim, Spark on Yarn allows us to do it using --num-executors and --executor_cores commandline arguments. I just got a chance to look at a similar spark user list mail, but no answer yet. So does mesos allow setting the number of executors and cores? Is there a default number it assumes?
On Mon, Jan 5, 2015 at 5:07 PM, Tim Chen <t...@mesosphere.io> wrote: > Forgot to hit reply-all. > > ---------- Forwarded message ---------- > From: Tim Chen <t...@mesosphere.io> > Date: Sun, Jan 4, 2015 at 10:46 PM > Subject: Re: Controlling number of executors on Mesos vs YARN > To: mvle <m...@us.ibm.com> > > > Hi Mike, > > You're correct there is no such setting in for Mesos coarse grain mode, > since the assumption is that each node is launched with one container and > Spark is launching multiple tasks in that container. > > In fine-grain mode there isn't a setting like that, as it currently will > launch an executor as long as it satisfies the minimum container resource > requirement. > > I've created a JIRA earlier about capping the number of executors or > better distribute the # of executors launched in each node. Since the > decision of choosing what node to launch containers is all in the Spark > scheduler side, it's very easy to modify it. > > Btw, what's the configuration to set the # of executors on YARN side? > > Thanks, > > Tim > > > > On Sun, Jan 4, 2015 at 9:37 PM, mvle <m...@us.ibm.com> wrote: > >> I'm trying to compare the performance of Spark running on Mesos vs YARN. >> However, I am having problems being able to configure the Spark workload >> to >> run in a similar way on Mesos and YARN. >> >> When running Spark on YARN, you can specify the number of executors per >> node. So if I have a node with 4 CPUs, I can specify 6 executors on that >> node. When running Spark on Mesos, there doesn't seem to be an equivalent >> way to specify this. In Mesos, you can somewhat force this by specifying >> the >> number of CPU resources to be 6 when running the slave daemon. However, >> this >> seems to be a static configuration of the Mesos cluster rather something >> that can be configured in the Spark framework. >> >> So here is my question: >> >> For Spark on Mesos, am I correct that there is no way to control the >> number >> of executors per node (assuming an idle cluster)? For Spark on Mesos >> coarse-grained mode, there is a way to specify max_cores but that is still >> not equivalent to specifying the number of executors per node as when >> Spark >> is run on YARN. >> >> If I am correct, then it seems Spark might be at a disadvantage running on >> Mesos compared to YARN (since it lacks the fine tuning ability provided by >> YARN). >> >> Thanks, >> Mike >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Controlling-number-of-executors-on-Mesos-vs-YARN-tp20966.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> > > -- Regards, Haripriya Ayyalasomayajula
