liyunzhang_intel created PIG-5068:
-------------------------------------
Summary: Set SPARK_REDUCERS by pig.properties not by system
configuration
Key: PIG-5068
URL: https://issues.apache.org/jira/browse/PIG-5068
Project: Pig
Issue Type: Bug
Reporter: liyunzhang_intel
Assignee: liyunzhang_intel
In SparkUtil.java, we set the SPARK_REDUCERS by system configuration
{code}
public static int getParallelism(List<RDD<Tuple>> predecessors,
PhysicalOperator physicalOperator) {
String numReducers = System.getenv("SPARK_REDUCERS");
if (numReducers != null) {
return Integer.parseInt(numReducers);
}
int parallelism = physicalOperator.getRequestedParallelism();
if (parallelism <= 0) {
// Parallelism wasn't set in Pig, so set it to whatever Spark thinks
// is reasonable.
parallelism = predecessors.get(0).context().defaultParallelism();
}
return parallelism;
}
{code}
It is better to set it by pig.properties
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
