[
https://issues.apache.org/jira/browse/SPARK-47001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Holden Karau updated SPARK-47001:
---------------------------------
Description: When pushing a filter down in a union the data type may not
match exactly if the filter was constructed using the child dataframe
reference. This is because the unions output is updated with a structype merge
of union which can turn non-nullable to nullable. These are still the same
column despite the different nullability so the filter should be safe to push
down. As it currently stands we get an exception. (was: Right now it asserts
exact equality but uses semanticEquality for candidacy, this can result in an
unexpected exception in Optimizer.scala when pushing down semantically equal
but different values.)
Summary: Pushdown Verification in Optimizer.scala should support
changed data types (was: Pushdown Verification in Optimizer.scala should use
semantic equals)
> Pushdown Verification in Optimizer.scala should support changed data types
> --------------------------------------------------------------------------
>
> Key: SPARK-47001
> URL: https://issues.apache.org/jira/browse/SPARK-47001
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: Holden Karau
> Assignee: Holden Karau
> Priority: Major
>
> When pushing a filter down in a union the data type may not match exactly if
> the filter was constructed using the child dataframe reference. This is
> because the unions output is updated with a structype merge of union which
> can turn non-nullable to nullable. These are still the same column despite
> the different nullability so the filter should be safe to push down. As it
> currently stands we get an exception.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
