[
https://issues.apache.org/jira/browse/PIG-4952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15731254#comment-15731254
]
liyunzhang_intel edited comment on PIG-4952 at 12/8/16 6:26 AM:
----------------------------------------------------------------
[~kexianda]:
"spark.default.parallism" is the configuration item to set the
parallism([see|http://spark.apache.org/docs/latest/configuration.html]). before
we use "spark.reducers", i guess "spark.reducers" is not official configuration
name. will change the name to "spark.default.parallism".
[~nkollar]:
{quote}
Does it make sense to set the parallelism level for these operators too,
{quote}
Make sense, will accept the modification in PIG-4952_1.patch about
SkewedJoinConverter.java,SortConverter.java and RankConverter.java.
[~kexianda] and [~nkollar]: please help review PIG-4952_2.patch in [review
board|https://reviews.apache.org/r/54523/]
was (Author: kellyzly):
[~kexianda]:
"spark.default.parallism" is the configuration item to set the
parallism([see|http://spark.apache.org/docs/latest/configuration.html]). before
we use "spark.reducers", i guess "spark.reducers" is not official configuration
name. will change the name to "spark.default.parallism".
[~nkollar]:
{quote}
Does it make sense to set the parallelism level for these operators too,
{quote}
Make sense, will accept the modification in PIG-4952_1.patch about
SkewedJoinConverter.java,SortConverter.java and RankConverter.java.
[~kexianda] and [~nkollar]: please help review PIG-4952_2.patch.
> Calculate the value of parallism for spark mode
> -----------------------------------------------
>
> Key: PIG-4952
> URL: https://issues.apache.org/jira/browse/PIG-4952
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: liyunzhang_intel
> Assignee: liyunzhang_intel
> Fix For: spark-branch
>
> Attachments: PIG-4952.patch, PIG-4952_1.patch, PIG-4952_2.patch
>
>
> Calculate the value of parallism for spark mode like what
> org.apache.pig.backend.hadoop.executionengine.tez.plan.optimizer.ParallelismSetter
> does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
