[
https://issues.apache.org/jira/browse/HIVE-8054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14135009#comment-14135009
]
Lefty Leverenz commented on HIVE-8054:
--------------------------------------
This should be documented in wikidoc "Configuration Properties" when the Spark
branch gets merged into trunk. In the meantime, it could be documented in
"Hive on Spark: Getting Started" as a note in the "Configuring Hive" section.
* [Hive on Spark: Getting Started -- Configuring Hive |
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive]
* [Configuration Properties -- hive.optimize.union.remove |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.union.remove]
> Disable hive.optimize.union.remove when hive.execution.engine=spark [Spark
> Branch]
> ----------------------------------------------------------------------------------
>
> Key: HIVE-8054
> URL: https://issues.apache.org/jira/browse/HIVE-8054
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Xuefu Zhang
> Assignee: Na Yang
> Labels: Spark-M1, TODOC-SPARK
> Fix For: spark-branch
>
> Attachments: HIVE-8054-spark.patch, HIVE-8054.2-spark.patch,
> HIVE-8054.3-spark.patch
>
>
> Option hive.optimize.union.remove introduced in HIVE-3276 removes union
> operators from the operator graph in certain cases as an optimization reduce
> the number of MR jobs. While making sense in MR, this optimization is
> actually harmful to an execution engine such as Spark, which natives supports
> union without requiring additional jobs. This is because removing union
> operator creates disjointed operator graphs, each graph generating a job, and
> thus this optimization requires more jobs to run the query. Not to mention
> the additional complexity handling linked FS descriptors.
> I propose that we disable such optimization when the execution engine is
> Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
