[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966587#comment-14966587
]
Srikanth Sundarrajan commented on PIG-4698:
-------------------------------------------
Am looking at propagating conf set in pig script to spark automatically such
that this feature can be enabled without too much translation inside pig for
approach #1. For instance the example below should do the trick.
{code}
set spark.dynamicAllocation.enabled true;
set spark.shuffle.service.enabled true;
A = LOAD '/tmp/in' USING PigStorage('\t') AS (line);
STORE A INTO '/tmp/out' USING PigStorage(',');
{code}
[~xuefuz], Thoughts ?
> Enable dynamic resource allocation/de-allocation on Yarn backends
> -----------------------------------------------------------------
>
> Key: PIG-4698
> URL: https://issues.apache.org/jira/browse/PIG-4698
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Affects Versions: spark-branch
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Labels: spork
> Fix For: spark-branch
>
>
> Resource elasticity needs to be enabled on Yarn backend to allow jobs to
> scale out better and provide better wall clock execution times, while unused
> resources should be released back to RM for use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
