[
https://issues.apache.org/jira/browse/SPARK-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zongheng Yang updated SPARK-2393:
-
Description:
Spark SQL should support the common optimization known as cost estimations. For
example, each logical operator should be able to estimate its cardinality,
based on the estimates from its children.
As a first step, the framework to support doing so should be added, which might
include the interface for the aforementioned cardinality estimation, and/or
some other metrics.
As the first proof of concept usage of this optimization, a simple optimization
strategy for certain equi-joins should be added: namely, if one side of a
qualifying join has a small estimated physical size (smaller than some
threshold), the planner should use a broadcast join physical plan to execute
the join, broadcasting the small side and streaming through the bigger side.
Similar concept exists in Hive and is explained
[here|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+JoinOptimization#LanguageManualJoinOptimization-OptimizeAutoJoinConversion].
Simple cost estimation and auto selection of broadcast join
---
Key: SPARK-2393
URL: https://issues.apache.org/jira/browse/SPARK-2393
Project: Spark
Issue Type: Bug
Components: SQL
Reporter: Michael Armbrust
Assignee: Zongheng Yang
Priority: Critical
Spark SQL should support the common optimization known as cost estimations.
For example, each logical operator should be able to estimate its
cardinality, based on the estimates from its children.
As a first step, the framework to support doing so should be added, which
might include the interface for the aforementioned cardinality estimation,
and/or some other metrics.
As the first proof of concept usage of this optimization, a simple
optimization strategy for certain equi-joins should be added: namely, if one
side of a qualifying join has a small estimated physical size (smaller than
some threshold), the planner should use a broadcast join physical plan to
execute the join, broadcasting the small side and streaming through the
bigger side. Similar concept exists in Hive and is explained
[here|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+JoinOptimization#LanguageManualJoinOptimization-OptimizeAutoJoinConversion].
--
This message was sent by Atlassian JIRA
(v6.2#6252)
