Yang Jie created SPARK-61872:
--------------------------------
Summary: Star schema detection does not recognize CHAR/VARCHAR
join keys
Key: SPARK-61872
URL: https://issues.apache.org/jira/browse/SPARK-61872
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
With {{spark.sql.cbo.starSchemaDetection=true}} (false by default),
{{StarSchemaDetection.findStarJoins}} finds no star join when the join keys are
CHAR columns. {{hasStatistics}} and {{isUnique}} look up the key's base column
through {{findLeafNodeCol}}, which passes a Project by looking for the same
attribute in the Project's child. The read-side padding Project from
SPARK-40697 (3.4.0) outputs {{readSidePadding(c, n) AS c}}, whose attribute has
a new exprId, so the base column is never found and the key counts as having no
statistics. The read-side checks of standard CHAR/VARCHAR semantics behave the
same way. When CHAR and VARCHAR are first-class types, a key compared with a
STRING key is also {{cast(c as string)}}, which the {{AttributeReference}}
patterns in {{findStarJoins}} do not match.
A catalyst probe on master with a fact table of 10000 rows (two keys, 100
values each) and two dimension tables of 100 rows (unique keys), all keys with
column statistics:
* STRING keys: a 3-table star join.
* CHAR(5) keys under read-side padding (default mode): none.
* CHAR(5) keys under the standard-semantics read-side check: none.
* The same keys cast to STRING: none.
{{findLeafNodeCol}} could see through {{CharVarcharUtils.ReadSidePadding}} and
{{ReadSideCheck}}, and the key patterns through
{{EstimationUtils.stripCastToPlainString}} from SPARK-60830.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]