[
https://issues.apache.org/jira/browse/PIG-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15728334#comment-15728334
]
Nandor Kollar commented on PIG-4952:
------------------------------------
[~kellyzly] you search for the max default parallelism value by iterating
through the parent RDDs, but since you use SparkContext, isn't this the same
for each parent RDD? How about this instead for tmpParallelism:
{code}
int tmpParallelism = predecessors.get(i).getNumPartitions();
{code}
How can we test that we achieved performance improvement?
> Calculate the value of parallism for spark mode
> -----------------------------------------------
>
> Key: PIG-4952
> URL: https://issues.apache.org/jira/browse/PIG-4952
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4952.patch, PIG-4952_1.patch
>
>
> Calculate the value of parallism for spark mode like what
> org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.ParallelismSetter
> does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
