`spark-submit` creates a new Application that will need to get resources from YARN. Spark's scheduler pools will determine how those resources are allocated among whatever Jobs run within the new Application.
Spark's scheduler pools are only relevant when you are submitting multiple Jobs within a single Application (i.e., you are using the same SparkContext to launch multiple Jobs) and you have used SparkContext#setLocalProperty to set "spark.scheduler.pool" to something other than the default pool before a particular Job intended to use that pool is started via that SparkContext. On Wed, Apr 5, 2017 at 1:11 PM, Nicholas Chammas <nicholas.cham...@gmail.com > wrote: > Hmm, so when I submit an application with `spark-submit`, I need to > guarantee it resources using YARN queues and not Spark's scheduler pools. > Is that correct? > > When are Spark's scheduler pools relevant/useful in this context? > > On Wed, Apr 5, 2017 at 3:54 PM Mark Hamstra <m...@clearstorydata.com> > wrote: > >> grrr... s/your/you're/ >> >> On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra <m...@clearstorydata.com> >> wrote: >> >> Your mixing up different levels of scheduling. Spark's fair scheduler >> pools are about scheduling Jobs, not Applications; whereas YARN queues with >> Spark are about scheduling Applications, not Jobs. >> >> On Wed, Apr 5, 2017 at 12:27 PM, Nick Chammas <nicholas.cham...@gmail.com >> > wrote: >> >> I'm having trouble understanding the difference between Spark fair >> scheduler pools >> <https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools> >> and YARN queues >> <https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>. >> Do they conflict? Does one override the other? >> >> I posted a more detailed question about an issue I'm having with this on >> Stack Overflow: http://stackoverflow.com/q/43239921/877069 >> >> Nick >> >> >> ------------------------------ >> View this message in context: Spark fair scheduler pools vs. YARN queues >> <http://apache-spark-user-list.1001560.n3.nabble.com/Spark-fair-scheduler-pools-vs-YARN-queues-tp28572.html> >> Sent from the Apache Spark User List mailing list archive >> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >> >> >> >>
