[
https://issues.apache.org/jira/browse/SPARK-60830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60830:
-----------------------------------
Labels: pull-request-available (was: )
> Keep CHAR/VARCHAR column statistics through read-side checks under standard
> semantics
> -------------------------------------------------------------------------------------
>
> Key: SPARK-60830
> URL: https://issues.apache.org/jira/browse/SPARK-60830
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> With spark.sql.charVarchar.standardSemantics.enabled=true,
> ApplyCharTypePadding wraps the CHAR columns of a scan in
> charTypeReadSideCheck and the VARCHAR columns in varcharTypeReadSideCheck
> instead of readSidePadding. CBO estimation does not recognize these calls, so
> the columns lose their statistics at the first Project and filters on them
> are estimated with selectivity 1.0. SPARK-60810 handles readSidePadding, the
> default setup, only.
> ANALYZE TABLE reads the table through the same checks, so the stored
> statistics describe the values the checks return. The checks need a matcher
> of their own: V2ExpressionBuilder uses the readSidePadding matcher to compile
> CHAR_READ_SIDE_PADDING, and that must not match the checks, which trim and
> fail on over-long values. The length floor that SPARK-60810 applies to padded
> CHAR values holds for charTypeReadSideCheck but not for
> varcharTypeReadSideCheck, whose values have at most N characters.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]