[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966876#comment-14966876
]
Srikanth Sundarrajan commented on PIG-4698:
-------------------------------------------
Yes. We need to enable auxillary shuffle service in YARN. Here is the list of
changes I had to do to get it to work.
1. Copied spark-1.4.1-yarn-shuffle.jar to yarn/lib folder on NodeManagers
2. Had to change the yarn-site on the NodeManager with the following changes
{noformat}
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle,spark_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.spark_shuffle.class</name>
<value>org.apache.spark.network.yarn.YarnShuffleService</value>
</property>
<property>
<name>spark.shuffle.service.enabled</name>
<value>true</value>
</property>
{noformat}
3. Restart the node manager
The gist of the change that you are likely to see in the patch is
{code}
//Copy all spark.* properties to SparkConf
for (String key : pigCtxtProperties.stringPropertyNames()) {
if (key.startsWith("spark.")) {
LOG.debug("Copying key " + key + " with value " +
pigCtxtProperties.getProperty(key) + " to SparkConf");
sparkConf.set(key, pigCtxtProperties.getProperty(key));
}
}
{code}
besides a few other changes & documentation.
I will finalize the patch and upload it sometime tomorrow. Also will verify
some scenarios where executors are removed and things fall back to aux-shuffle
service works fine.
> Enable dynamic resource allocation/de-allocation on Yarn backends
> -----------------------------------------------------------------
>
> Key: PIG-4698
> URL: https://issues.apache.org/jira/browse/PIG-4698
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Affects Versions: spark-branch
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Labels: spork
> Fix For: spark-branch
>
>
> Resource elasticity needs to be enabled on Yarn backend to allow jobs to
> scale out better and provide better wall clock execution times, while unused
> resources should be released back to RM for use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
