I encountered the following issue when enabling dynamicAllocation. You may want to take a look at it.
https://issues.apache.org/jira/browse/SPARK-4951 Best Regards, Shixiong Zhu 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA <ozawa.tsuyo...@gmail.com>: > Hi Anders, > > I faced the same issue as you mentioned. Yes, you need to install > spark shuffle plugin for YARN. Please check following PRs which add > doc to enable dynamicAllocation: > > https://github.com/apache/spark/pull/3731 > https://github.com/apache/spark/pull/3757 > > I could run Spark on YARN with dynamicAllocation by following the > instructions described in the docs. > > Thanks, > - Tsuyoshi > > On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg <arp...@spotify.com> > wrote: > > Hey, > > > > Tried to get the new spark.dynamicAllocation.enabled feature working on > Yarn > > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following > > settings: > > > > conf > > .set("spark.dynamicAllocation.enabled", "true") > > .set("spark.shuffle.service.enabled", "true") > > .set("spark.dynamicAllocation.minExecutors", "10") > > .set("spark.dynamicAllocation.maxExecutors", "700") > > > > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but > > with the settings above, it will start the app and the first job is > listed > > in the web ui. However, no tasks are started and it seems to be stuck > > waiting for a container to be allocated forever. > > > > Any help would be appreciated. Need to do something specific to get the > > external yarn shuffle service running in the node manager? > > > > TIA, > > Anders > > > > -- > - Tsuyoshi > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
