Sebastian created SPARK-7277:
--------------------------------
Summary: property mapred.reduce.task replaced by
spark.sql.shuffle.partitions
Key: SPARK-7277
URL: https://issues.apache.org/jira/browse/SPARK-7277
Project: Spark
Issue Type: Bug
Affects Versions: 1.3.1
Reporter: Sebastian
When I use "SET mapred.reduce.task" I get the warning "SetCommand: Property
mapred.reduce.tasks is deprecated, automatically converted to
spark.sql.shuffle.partitions instead."
It's true that mapred.reduce.task is deprecated but this replacement causes
serious trouble:
Setting mapred.reduce.task to -1 (negative one) is valid and causes hadoop/hive
to automatically determine the required number of reducers.
Setting spark.sql.shuffle.partitions to negative can cause spark to produce
incorrect results.
In my system (spark-sql 1.3.1 running on a single machine in "local" mode),
with this setting, any outer join produces no output (whereas an inner join
does)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
