[
https://issues.apache.org/jira/browse/SPARK-48360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-48360:
-----------------------------------
Labels: pull-request-available (was: )
> Simplify conditionals with predicate branches
> ---------------------------------------------
>
> Key: SPARK-48360
> URL: https://issues.apache.org/jira/browse/SPARK-48360
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.4.3
> Reporter: Thomas Powell
> Priority: Major
> Labels: pull-request-available
>
> The catalyst optimizer has many optimizations for {{CaseWhen}} and {{If}}
> expressions, that can eliminate branches entirely or replace them with
> Boolean logic. There are additional "always false" conditionals that could be
> eliminated entirely. It would also be possible to replace conditionals with
> Boolean logic where the {{if-branch}} and {{else-branch}} are themselves
> predicates. The primary motivation would be to push more filters to the
> datasource.
> For example:
> {code:java}
> Filter(If(GreaterThan(a, 2), false, LessThanOrEqual(b <= 4))){code}
> is equivalent to
> {code:java}
> # a not nullable
> Filter(And(LessThanOrEqual(a, 2), LessThanOrEqual(b, 4))
> # a nullable
> Filter(And(Not(EqualNotSafe(GreaterThan(a, 2), true), LessThanOrEqual(b,
> 4)))) {code}
> Within a filter the nullability handling is admittedly less important since
> the expression evaluating to null would be semantically equivalent to false,
> but the original conditional may have been intentionally written to not
> return null when {{a}} may be null.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
