okumin created HIVE-23831:
-----------------------------
Summary: Enforce auto-parallelism of Tez even if the number of
reducers can be 1
Key: HIVE-23831
URL: https://issues.apache.org/jira/browse/HIVE-23831
Project: Hive
Issue Type: Improvement
Components: Hive
Reporter: okumin
Assignee: okumin
Hive on Tez disables auto-reduce and uses the maximum number of partitions when
the minimum number of reducers, `estimated number * min.partition.factor`, is
less than or equal to 1.
[https://github.com/apache/hive/blob/f98e136bdd5642e3de10d2fd1a4c14d1d6762113/ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java#L121-L131]
https://issues.apache.org/jira/browse/HIVE-14200
This can make Tez lose chance to optimize # of reducers especially in an
environment with small `hive.tez.min.partition.factor`.
I'd say it would be fine to introduce an option to enforce auto-reduce.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
