[
https://issues.apache.org/jira/browse/HIVE-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhenxiao Luo reassigned HIVE-3733:
--
Assignee: Zhenxiao Luo (was: Pradeep Kamath)
Improve Hive's logic for conditional merge
--
Key: HIVE-3733
URL: https://issues.apache.org/jira/browse/HIVE-3733
Project: Hive
Issue Type: Improvement
Reporter: Pradeep Kamath
Assignee: Zhenxiao Luo
Attachments: HIVE-3733.1.patch.txt, HIVE-3733.3.patch.txt,
HIVE-3733.4.patch.txt, HIVE-3733.5.patch.txt, HIVE-3733.optimizer.patch.txt
If the config hive.merge.mapfiles is set to true and hive.merge.mapredfiles
is set to false then when hive encounters a FileSinkOperator when generating
map reduce tasks, it will look at the entire job to see if it has a reducer,
if it does it will not merge. Instead it should be check if the
FileSinkOperator is a child of the reducer. This means that outputs generated
in the mapper will be merged, and outputs generated in the reducer will not
be, the intended effect of setting those configs.
Simple repro:
set hive.merge.mapfiles=true;
set hive.merge.mapredfiles=false;
EXPLAIN
FROM input_table
INSERT OVERWRITE TABLE output_table1 SELECT key, COUNT(*) group by key
INSERT OVERWRITE TABLE output_table2 SELECT *;
The output should contain a Conditional Operator, Mapred Stages, and Move
tasks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
