[ https://issues.apache.org/jira/browse/PIG-5068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
liyunzhang_intel updated PIG-5068: ---------------------------------- Attachment: (was: PIG-5068_1.patch) > Set SPARK_REDUCERS by pig.properties not by system configuration > ---------------------------------------------------------------- > > Key: PIG-5068 > URL: https://issues.apache.org/jira/browse/PIG-5068 > Project: Pig > Issue Type: Sub-task > Components: spark > Reporter: liyunzhang_intel > Assignee: liyunzhang_intel > Attachments: PIG-5068.patch, PIG-5068_1.patch > > > In SparkUtil.java, we set the SPARK_REDUCERS by system configuration > {code} > public static int getParallelism(List<RDD<Tuple>> predecessors, > PhysicalOperator physicalOperator) { > String numReducers = System.getenv("SPARK_REDUCERS"); > if (numReducers != null) { > return Integer.parseInt(numReducers); > } > int parallelism = physicalOperator.getRequestedParallelism(); > if (parallelism <= 0) { > // Parallelism wasn't set in Pig, so set it to whatever Spark > thinks > // is reasonable. > parallelism = predecessors.get(0).context().defaultParallelism(); > } > return parallelism; > } > {code} > It is better to set it by pig.properties -- This message was sent by Atlassian JIRA (v6.3.4#6332)