Hey Arun, The sleeps would only cause maximum like 5 second overhead. The idea was to give executors some time to register. On more recent versions, they were replaced with the spark.scheduler.minRegisteredResourcesRatio and spark.scheduler.maxRegisteredResourcesWaitingTime. As of 1.1, by default YARN will wait until either 30 seconds have passed or 80% of the requested executors have registered.
-Sandy On Fri, Dec 5, 2014 at 12:46 PM, Ashish Rangole <arang...@gmail.com> wrote: > Likely this not the case here yet one thing to point out with Yarn > parameters like --num-executors is that they should be specified *before* > app jar and app args on spark-submit command line otherwise the app only > gets the default number of containers which is 2. > On Dec 5, 2014 12:22 PM, "Sandy Ryza" <sandy.r...@cloudera.com> wrote: > >> Hi Denny, >> >> Those sleeps were only at startup, so if jobs are taking significantly >> longer on YARN, that should be a different problem. When you ran on YARN, >> did you use the --executor-cores, --executor-memory, and --num-executors >> arguments? When running against a standalone cluster, by default Spark >> will make use of all the cluster resources, but when running against YARN, >> Spark defaults to a couple tiny executors. >> >> -Sandy >> >> On Fri, Dec 5, 2014 at 11:32 AM, Denny Lee <denny.g....@gmail.com> wrote: >> >>> My submissions of Spark on YARN (CDH 5.2) resulted in a few thousand >>> steps. If I was running this on standalone cluster mode the query finished >>> in 55s but on YARN, the query was still running 30min later. Would the hard >>> coded sleeps potentially be in play here? >>> On Fri, Dec 5, 2014 at 11:23 Sandy Ryza <sandy.r...@cloudera.com> wrote: >>> >>>> Hi Tobias, >>>> >>>> What version are you using? In some recent versions, we had a couple >>>> of large hardcoded sleeps on the Spark side. >>>> >>>> -Sandy >>>> >>>> On Fri, Dec 5, 2014 at 11:15 AM, Andrew Or <and...@databricks.com> >>>> wrote: >>>> >>>>> Hey Tobias, >>>>> >>>>> As you suspect, the reason why it's slow is because the resource >>>>> manager in YARN takes a while to grant resources. This is because YARN >>>>> needs to first set up the application master container, and then this AM >>>>> needs to request more containers for Spark executors. I think this >>>>> accounts >>>>> for most of the overhead. The remaining source probably comes from how our >>>>> own YARN integration code polls application (every second) and cluster >>>>> resource states (every 5 seconds IIRC). I haven't explored in detail >>>>> whether there are optimizations there that can speed this up, but I >>>>> believe >>>>> most of the overhead comes from YARN itself. >>>>> >>>>> In other words, no I don't know of any quick fix on your end that you >>>>> can do to speed this up. >>>>> >>>>> -Andrew >>>>> >>>>> >>>>> 2014-12-03 20:10 GMT-08:00 Tobias Pfeiffer <t...@preferred.jp>: >>>>> >>>>> Hi, >>>>>> >>>>>> I am using spark-submit to submit my application to YARN in >>>>>> "yarn-cluster" mode. I have both the Spark assembly jar file as well as >>>>>> my >>>>>> application jar file put in HDFS and can see from the logging output that >>>>>> both files are used from there. However, it still takes about 10 seconds >>>>>> for my application's yarnAppState to switch from ACCEPTED to RUNNING. >>>>>> >>>>>> I am aware that this is probably not a Spark issue, but some YARN >>>>>> configuration setting (or YARN-inherent slowness), I was just wondering >>>>>> if >>>>>> anyone has an advice for how to speed this up. >>>>>> >>>>>> Thanks >>>>>> Tobias >>>>>> >>>>> >>>>> >>>> >>
