[
https://issues.apache.org/jira/browse/PIG-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14966590#comment-14966590
]
Srikanth Sundarrajan commented on PIG-4698:
-------------------------------------------
Or more tersely
{code}
set spark.dynamicAllocation.enabled true;
A = LOAD '/tmp/in' USING PigStorage('\t') AS (line);
STORE A INTO '/tmp/out' USING PigStorage(',');
{code}
dropping {{set spark.shuffle.service.enabled true;}}, as the same can be
automatically set if dynamic is enabled in YARN mode.
> Enable dynamic resource allocation/de-allocation on Yarn backends
> -----------------------------------------------------------------
>
> Key: PIG-4698
> URL: https://issues.apache.org/jira/browse/PIG-4698
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Affects Versions: spark-branch
> Reporter: Srikanth Sundarrajan
> Assignee: Srikanth Sundarrajan
> Labels: spork
> Fix For: spark-branch
>
>
> Resource elasticity needs to be enabled on Yarn backend to allow jobs to
> scale out better and provide better wall clock execution times, while unused
> resources should be released back to RM for use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
