Yuming Wang created SPARK-19693: ----------------------------------- Summary: SET mapreduce.job.reduces automatically converted to spark.sql.shuffle.partitions Key: SPARK-19693 URL: https://issues.apache.org/jira/browse/SPARK-19693 Project: Spark Issue Type: Improvement Components: SQL Affects Versions: 2.1.0, 2.2.0 Reporter: Yuming Wang
How to reproduce: {code} spark-sql> set spark.sql.shuffle.partitions; spark.sql.shuffle.partitions 200 Time taken: 7.118 seconds, Fetched 1 row(s) spark-sql> set mapred.reduce.tasks=10; spark.sql.shuffle.partitions 10 Time taken: 5.968 seconds, Fetched 1 row(s) spark-sql> set mapreduce.job.reduces=20; set mapreduce.job.reduces 20 Time taken: 0.233 seconds, Fetched 1 row(s) spark-sql> set spark.sql.shuffle.partitions; spark.sql.shuffle.partitions 10 Time taken: 0.099 seconds, Fetched 1 row(s) {code} The {{SET mapreduce.job.reduces}} should automatically converted to {{spark.sql.shuffle.partitions}}, it's similar to {{set mapred.reduce.tasks}}. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org