Thanks Tsuyoshi and Shixiong for the info. Awesome with more documentation about the feature!
Was afraid that the node manager needed reconfiguration (and restart). Any idea of how much resources will the shuffle service take on the node manager? In a multi-tenant Hadoop cluster environment, it would be undesirable to have a Spark-specific long running service taking up resources from other types of jobs on the cluster. Thanks again, Anders On Sun, Dec 28, 2014 at 8:08 AM, Shixiong Zhu <zsxw...@gmail.com> wrote: > I encountered the following issue when enabling dynamicAllocation. You may > want to take a look at it. > > https://issues.apache.org/jira/browse/SPARK-4951 > > Best Regards, > Shixiong Zhu > > 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA <ozawa.tsuyo...@gmail.com>: > >> Hi Anders, >> >> I faced the same issue as you mentioned. Yes, you need to install >> spark shuffle plugin for YARN. Please check following PRs which add >> doc to enable dynamicAllocation: >> >> https://github.com/apache/spark/pull/3731 >> https://github.com/apache/spark/pull/3757 >> >> I could run Spark on YARN with dynamicAllocation by following the >> instructions described in the docs. >> >> Thanks, >> - Tsuyoshi >> >> On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg <arp...@spotify.com> >> wrote: >> > Hey, >> > >> > Tried to get the new spark.dynamicAllocation.enabled feature working on >> Yarn >> > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following >> > settings: >> > >> > conf >> > .set("spark.dynamicAllocation.enabled", "true") >> > .set("spark.shuffle.service.enabled", "true") >> > .set("spark.dynamicAllocation.minExecutors", "10") >> > .set("spark.dynamicAllocation.maxExecutors", "700") >> > >> > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but >> > with the settings above, it will start the app and the first job is >> listed >> > in the web ui. However, no tasks are started and it seems to be stuck >> > waiting for a container to be allocated forever. >> > >> > Any help would be appreciated. Need to do something specific to get the >> > external yarn shuffle service running in the node manager? >> > >> > TIA, >> > Anders >> >> >> >> -- >> - Tsuyoshi >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
