Yang Jie created SPARK-61872:
--------------------------------

             Summary: Star schema detection does not recognize CHAR/VARCHAR 
join keys
                 Key: SPARK-61872
                 URL: https://issues.apache.org/jira/browse/SPARK-61872
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Yang Jie


With {{spark.sql.cbo.starSchemaDetection=true}} (false by default), 
{{StarSchemaDetection.findStarJoins}} finds no star join when the join keys are 
CHAR columns. {{hasStatistics}} and {{isUnique}} look up the key's base column 
through {{findLeafNodeCol}}, which passes a Project by looking for the same 
attribute in the Project's child. The read-side padding Project from 
SPARK-40697 (3.4.0) outputs {{readSidePadding(c, n) AS c}}, whose attribute has 
a new exprId, so the base column is never found and the key counts as having no 
statistics. The read-side checks of standard CHAR/VARCHAR semantics behave the 
same way. When CHAR and VARCHAR are first-class types, a key compared with a 
STRING key is also {{cast(c as string)}}, which the {{AttributeReference}} 
patterns in {{findStarJoins}} do not match.

A catalyst probe on master with a fact table of 10000 rows (two keys, 100 
values each) and two dimension tables of 100 rows (unique keys), all keys with 
column statistics:
* STRING keys: a 3-table star join.
* CHAR(5) keys under read-side padding (default mode): none.
* CHAR(5) keys under the standard-semantics read-side check: none.
* The same keys cast to STRING: none.

{{findLeafNodeCol}} could see through {{CharVarcharUtils.ReadSidePadding}} and 
{{ReadSideCheck}}, and the key patterns through 
{{EstimationUtils.stripCastToPlainString}} from SPARK-60830.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to