Yang Jie created SPARK-60830:
--------------------------------
Summary: Keep CHAR/VARCHAR column statistics through read-side
checks under standard semantics
Key: SPARK-60830
URL: https://issues.apache.org/jira/browse/SPARK-60830
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
With spark.sql.charVarchar.standardSemantics.enabled=true, ApplyCharTypePadding
wraps the CHAR columns of a scan in charTypeReadSideCheck and the VARCHAR
columns in varcharTypeReadSideCheck instead of readSidePadding. CBO estimation
does not recognize these calls, so the columns lose their statistics at the
first Project and filters on them are estimated with selectivity 1.0.
SPARK-60810 handles readSidePadding, the default setup, only.
ANALYZE TABLE reads the table through the same checks, so the stored statistics
describe the values the checks return. The checks need a matcher of their own:
V2ExpressionBuilder uses the readSidePadding matcher to compile
CHAR_READ_SIDE_PADDING, and that must not match the checks, which trim and fail
on over-long values. The length floor that SPARK-60810 applies to padded CHAR
values holds for charTypeReadSideCheck but not for varcharTypeReadSideCheck,
whose values have at most N characters.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
