Hey Tobias,
Can you try using the YARN Fair Scheduler and set
yarn.scheduler.fair.continuous-scheduling-enabled to true?
-Sandy
On Sun, Dec 7, 2014 at 5:39 PM, Tobias Pfeiffer t...@preferred.jp wrote:
Hi,
thanks for your responses!
On Sat, Dec 6, 2014 at 4:22 AM, Sandy Ryza
Hi,
On Tue, Dec 9, 2014 at 4:39 AM, Sandy Ryza sandy.r...@cloudera.com wrote:
Can you try using the YARN Fair Scheduler and set
yarn.scheduler.fair.continuous-scheduling-enabled to true?
I'm using Cloudera 5.2.0 and my configuration says
yarn.resourcemanager.scheduler.class =
Hi,
thanks for your responses!
On Sat, Dec 6, 2014 at 4:22 AM, Sandy Ryza sandy.r...@cloudera.com wrote:
What version are you using? In some recent versions, we had a couple of
large hardcoded sleeps on the Spark side.
I am using Spark 1.1.1.
As Andrew mentioned, I guess most of the 10
Great to hear!
-Sandy
On Fri, Dec 5, 2014 at 11:17 PM, Denny Lee denny.g@gmail.com wrote:
Okay, my bad for not testing out the documented arguments - once i use the
correct ones, the query shrinks completes in ~55s (I can probably make it
faster). Thanks for the help, eh?!
On Fri
Hey Tobias,
As you suspect, the reason why it's slow is because the resource manager in
YARN takes a while to grant resources. This is because YARN needs to first
set up the application master container, and then this AM needs to request
more containers for Spark executors. I think this accounts
Hi Tobias,
What version are you using? In some recent versions, we had a couple of
large hardcoded sleeps on the Spark side.
-Sandy
On Fri, Dec 5, 2014 at 11:15 AM, Andrew Or and...@databricks.com wrote:
Hey Tobias,
As you suspect, the reason why it's slow is because the resource manager
My submissions of Spark on YARN (CDH 5.2) resulted in a few thousand steps.
If I was running this on standalone cluster mode the query finished in 55s
but on YARN, the query was still running 30min later. Would the hard coded
sleeps potentially be in play here?
On Fri, Dec 5, 2014 at 11:23 Sandy
Just an FYI - I can submit the SparkPi app to YARN in cluster mode on a
1-node m3.xlarge EC2 instance instance and the app finishes running
successfully in about 40 seconds. I just figured the 30 - 40 sec run time
was normal b/c of the submitting overhead that Andrew mentioned.
Denny, you can
Hi Denny,
Those sleeps were only at startup, so if jobs are taking significantly
longer on YARN, that should be a different problem. When you ran on YARN,
did you use the --executor-cores, --executor-memory, and --num-executors
arguments? When running against a standalone cluster, by default
Hey Sandy,
What are those sleeps for and do they still exist? We have seen about a
1min to 1:30 executor startup time, which is a large chunk for jobs that
run in ~10min.
Thanks,
Arun
On Fri, Dec 5, 2014 at 3:20 PM, Sandy Ryza sandy.r...@cloudera.com wrote:
Hi Denny,
Those sleeps were only
Likely this not the case here yet one thing to point out with Yarn
parameters like --num-executors is that they should be specified *before*
app jar and app args on spark-submit command line otherwise the app only
gets the default number of containers which is 2.
On Dec 5, 2014 12:22 PM, Sandy
Hey Arun,
The sleeps would only cause maximum like 5 second overhead. The idea was
to give executors some time to register. On more recent versions, they
were replaced with the spark.scheduler.minRegisteredResourcesRatio and
spark.scheduler.maxRegisteredResourcesWaitingTime. As of 1.1, by
Sorry for the delay in my response - for my spark calls for stand-alone and
YARN, I am using the --executor-memory and --total-executor-cores for the
submission. In standalone, my baseline query completes in ~40s while in
YARN, it completes in ~1800s. It does not appear from the RM web UI that
Okay, my bad for not testing out the documented arguments - once i use the
correct ones, the query shrinks completes in ~55s (I can probably make it
faster). Thanks for the help, eh?!
On Fri Dec 05 2014 at 10:34:50 PM Denny Lee denny.g@gmail.com wrote:
Sorry for the delay in my response
14 matches
Mail list logo
