[
https://issues.apache.org/jira/browse/HIVE-8054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14133227#comment-14133227
]
Xuefu Zhang commented on HIVE-8054:
-----------------------------------
Hi [~nyang], thank you very much for working on this. The patch looks good, and
I just submitted it to let the test run.
+1 pending on test result.
> Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark
> Branch]
> ----------------------------------------------------------------------------------
>
> Key: HIVE-8054
> URL: https://issues.apache.org/jira/browse/HIVE-8054
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Na Yang
> Labels: Spark-M1
> Attachments: HIVE-8054-spark.patch
>
>
> Option hive.optimize.union.remove introduced in HIVE-3276 removes union
> operators from the operator graph in certain cases as an optimization reduce
> the number of MR jobs. While making sense in MR, this optimization is
> actually harmful to an execution engine such as Spark, which natives supports
> union without requiring additional jobs. This is because removing union
> operator creates disjointed operator graphs, each graph generating a job, and
> thus this optimization requires more jobs to run the query. Not to mention
> the additional complexity handling linked FS descriptors.
> I propose that we disable such optimization when the execution engine is
> Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
