Re: Dynamic Allocation in Spark 1.2.0
Thanks Tsuyoshi and Shixiong for the info. Awesome with more documentation about the feature! Was afraid that the node manager needed reconfiguration (and restart). Any idea of how much resources will the shuffle service take on the node manager? In a multi-tenant Hadoop cluster environment, it would be undesirable to have a Spark-specific long running service taking up resources from other types of jobs on the cluster. Thanks again, Anders On Sun, Dec 28, 2014 at 8:08 AM, Shixiong Zhu zsxw...@gmail.com wrote: I encountered the following issue when enabling dynamicAllocation. You may want to take a look at it. https://issues.apache.org/jira/browse/SPARK-4951 Best Regards, Shixiong Zhu 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com: Hi Anders, I faced the same issue as you mentioned. Yes, you need to install spark shuffle plugin for YARN. Please check following PRs which add doc to enable dynamicAllocation: https://github.com/apache/spark/pull/3731 https://github.com/apache/spark/pull/3757 I could run Spark on YARN with dynamicAllocation by following the instructions described in the docs. Thanks, - Tsuyoshi On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg arp...@spotify.com wrote: Hey, Tried to get the new spark.dynamicAllocation.enabled feature working on Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the following settings: conf .set(spark.dynamicAllocation.enabled, true) .set(spark.shuffle.service.enabled, true) .set(spark.dynamicAllocation.minExecutors, 10) .set(spark.dynamicAllocation.maxExecutors, 700) The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but with the settings above, it will start the app and the first job is listed in the web ui. However, no tasks are started and it seems to be stuck waiting for a container to be allocated forever. Any help would be appreciated. Need to do something specific to get the external yarn shuffle service running in the node manager? TIA, Anders -- - Tsuyoshi - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Dynamic Allocation in Spark 1.2.0
Hey, Tried to get the new spark.dynamicAllocation.enabled feature working on Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the following settings: conf .set(spark.dynamicAllocation.enabled, true) .set(spark.shuffle.service.enabled, true) .set(spark.dynamicAllocation.minExecutors, 10) .set(spark.dynamicAllocation.maxExecutors, 700) The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but with the settings above, it will start the app and the first job is listed in the web ui. However, no tasks are started and it seems to be stuck waiting for a container to be allocated forever. Any help would be appreciated. Need to do something specific to get the external yarn shuffle service running in the node manager? TIA, Anders
Dynamic Allocation in Spark 1.2.0
Hey, Tried to get the new spark.dynamicAllocation.enabled feature working on Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the following settings: conf .set(spark.dynamicAllocation.enabled, true) .set(spark.shuffle.service.enabled, true) .set(spark.dynamicAllocation.minExecutors, 10) .set(spark.dynamicAllocation.maxExecutors, 700) The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but with the settings above, it will start the app and the first job is listed in the web ui. However, no tasks are started and it seems to be stuck waiting for a container to be allocated forever. Any help would be appreciated. Need to do something specific to get the external yarn shuffle service running in the node manager? TIA, Anders
Re: Dynamic Allocation in Spark 1.2.0
Hi Anders, I faced the same issue as you mentioned. Yes, you need to install spark shuffle plugin for YARN. Please check following PRs which add doc to enable dynamicAllocation: https://github.com/apache/spark/pull/3731 https://github.com/apache/spark/pull/3757 I could run Spark on YARN with dynamicAllocation by following the instructions described in the docs. Thanks, - Tsuyoshi On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg arp...@spotify.com wrote: Hey, Tried to get the new spark.dynamicAllocation.enabled feature working on Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the following settings: conf .set(spark.dynamicAllocation.enabled, true) .set(spark.shuffle.service.enabled, true) .set(spark.dynamicAllocation.minExecutors, 10) .set(spark.dynamicAllocation.maxExecutors, 700) The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but with the settings above, it will start the app and the first job is listed in the web ui. However, no tasks are started and it seems to be stuck waiting for a container to be allocated forever. Any help would be appreciated. Need to do something specific to get the external yarn shuffle service running in the node manager? TIA, Anders -- - Tsuyoshi - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Dynamic Allocation in Spark 1.2.0
I encountered the following issue when enabling dynamicAllocation. You may want to take a look at it. https://issues.apache.org/jira/browse/SPARK-4951 Best Regards, Shixiong Zhu 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com: Hi Anders, I faced the same issue as you mentioned. Yes, you need to install spark shuffle plugin for YARN. Please check following PRs which add doc to enable dynamicAllocation: https://github.com/apache/spark/pull/3731 https://github.com/apache/spark/pull/3757 I could run Spark on YARN with dynamicAllocation by following the instructions described in the docs. Thanks, - Tsuyoshi On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg arp...@spotify.com wrote: Hey, Tried to get the new spark.dynamicAllocation.enabled feature working on Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the following settings: conf .set(spark.dynamicAllocation.enabled, true) .set(spark.shuffle.service.enabled, true) .set(spark.dynamicAllocation.minExecutors, 10) .set(spark.dynamicAllocation.maxExecutors, 700) The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but with the settings above, it will start the app and the first job is listed in the web ui. However, no tasks are started and it seems to be stuck waiting for a container to be allocated forever. Any help would be appreciated. Need to do something specific to get the external yarn shuffle service running in the node manager? TIA, Anders -- - Tsuyoshi - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
