[
https://issues.apache.org/jira/browse/HIVE-3733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated HIVE-3733:
---------------------------------
Attachment: HIVE-3733.optimizer.patch.txt
Uploading a new version (attached as HIVE-3733.optimizer.patch.txt, I have also
updated the phabricator review with this new code) based on new logic to work
at the physical optimizer level.
- I am still running the tests - this is an early preview so I know that I am
on the right track
- I feel like the wiring between stages is not quite right though I couldn't
figure out enough to know - the creation of the ConditionalTask is pretty deep
into the guts of task/operator code which I am not very familiar. For example,
in the new union19.q.out, one of the stages (stage-6) is missing
There maybe room for improving the code - I wanted early feedback - so please
do review the code carefully.
> Improve Hive's logic for conditional merge
> ------------------------------------------
>
> Key: HIVE-3733
> URL: https://issues.apache.org/jira/browse/HIVE-3733
> Project: Hive
> Issue Type: Improvement
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Attachments: HIVE-3733.1.patch.txt, HIVE-3733.3.patch.txt,
> HIVE-3733.4.patch.txt, HIVE-3733.optimizer.patch.txt
>
>
> If the config hive.merge.mapfiles is set to true and hive.merge.mapredfiles
> is set to false then when hive encounters a FileSinkOperator when generating
> map reduce tasks, it will look at the entire job to see if it has a reducer,
> if it does it will not merge. Instead it should be check if the
> FileSinkOperator is a child of the reducer. This means that outputs generated
> in the mapper will be merged, and outputs generated in the reducer will not
> be, the intended effect of setting those configs.
> Simple repro:
> set hive.merge.mapfiles=true;
> set hive.merge.mapredfiles=false;
> EXPLAIN
> FROM <input_table>
> INSERT OVERWRITE TABLE <output_table1> SELECT key, COUNT(*) group by key
> INSERT OVERWRITE TABLE <output_table2> SELECT *;
> The output should contain a Conditional Operator, Mapred Stages, and Move
> tasks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
