ns that I forgot your typical Spark application submits multiple
"jobs", each of which has multiple stages, etc.
So in my case I need to read up more closely about YARN queues
<https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>
since I want to
parkContext.
On Wed, Apr 5, 2017 at 1:11 PM, Nicholas Chammas wrote:
> Hmm, so when I submit an application with `spark-submit`, I need to
> guarantee it resources using YARN queues and not Spark's scheduler pools.
> Is that correct?
>
> When are Spark's scheduler pools rel
Hmm, so when I submit an application with `spark-submit`, I need to
guarantee it resources using YARN queues and not Spark's scheduler pools.
Is that correct?
When are Spark's scheduler pools relevant/useful in this context?
On Wed, Apr 5, 2017 at 3:54 PM Mark Hamstra wrote:
> g
grrr... s/your/you're/
On Wed, Apr 5, 2017 at 12:54 PM, Mark Hamstra
wrote:
> Your mixing up different levels of scheduling. Spark's fair scheduler
> pools are about scheduling Jobs, not Applications; whereas YARN queues with
> Spark are about scheduling Applications, not Jo
Your mixing up different levels of scheduling. Spark's fair scheduler pools
are about scheduling Jobs, not Applications; whereas YARN queues with Spark
are about scheduling Applications, not Jobs.
On Wed, Apr 5, 2017 at 12:27 PM, Nick Chammas
wrote:
> I'm having trouble unde
I'm having trouble understanding the difference between Spark fair
scheduler pools
<https://spark.apache.org/docs/latest/job-scheduling.html#fair-scheduler-pools>
and YARN queues
<https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html>.
Do they c
Hi,
I'm trying to get my head around the different parts of Spark on YARN
architecture with YARN's schedulers and queues as well as Spark's own
schedulers - FAIR and FIFO.
I'd appreciate if you could read how I see things and correct me where
I'm wrong. Thanks!
The def
Has anyone implemented Queues using RDDs?
Thank You