Thanks Tsuyoshi and Shixiong for the info. Awesome with more documentation
about the feature!
Was afraid that the node manager needed reconfiguration (and restart). Any
idea of how much resources will the shuffle service take on the node
manager? In a multi-tenant Hadoop cluster environment, it
Hey,
Tried to get the new spark.dynamicAllocation.enabled feature working on
Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the
following settings:
conf
.set(spark.dynamicAllocation.enabled, true)
.set(spark.shuffle.service.enabled, true)
Hey,
Tried to get the new spark.dynamicAllocation.enabled feature working on
Yarn (Hadoop 2.2), but am unsuccessful so far. I've tested with the
following settings:
conf
.set(spark.dynamicAllocation.enabled, true)
.set(spark.shuffle.service.enabled, true)
Hi Anders,
I faced the same issue as you mentioned. Yes, you need to install
spark shuffle plugin for YARN. Please check following PRs which add
doc to enable dynamicAllocation:
https://github.com/apache/spark/pull/3731
https://github.com/apache/spark/pull/3757
I could run Spark on YARN with
I encountered the following issue when enabling dynamicAllocation. You may
want to take a look at it.
https://issues.apache.org/jira/browse/SPARK-4951
Best Regards,
Shixiong Zhu
2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com:
Hi Anders,
I faced the same issue as you
