[
https://issues.apache.org/jira/browse/SPARK-60602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-60602:
-----------------------------------
Assignee: L. C. Hsieh
> Do not evaluate throwable filter conditions ahead of the conditions before
> them when pushing filters through a projection
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60602
> URL: https://issues.apache.org/jira/browse/SPARK-60602
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0, 4.2.1, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
>
> PushPredicateThroughNonJoin can evaluate a throwable filter condition
> (raise_error / assert_true) on rows that the conditions written before it
> would have filtered out, so a query that succeeds without the reordering
> fails with USER_RAISED_EXCEPTION.
> There are two paths that ignore Expression.throwable when reordering the
> conjuncts of a Filter over a Project with expensive (UDF, expensive regex)
> aliases:
> 1. The projection split added by SPARK-55014
> (spark.sql.optimizer.splitProjectionForExpensiveFilters, default true) orders
> condition groups by how many not-yet-computed aliases they need. With f and g
> expensive UDFs:
> {code}
> SELECT * FROM (SELECT id, f(id) AS f, g(id) AS g FROM range(10))
> WHERE f + g > 10 AND assert_true(g > 5) IS NULL
> {code}
> The assert_true group needs fewer aliases, so it is split below the
> projection and evaluated on ids 0..5, which the AND short-circuit would have
> skipped.
> 2. The cheap/expensive partition added by SPARK-47672
> (spark.sql.optimizer.avoidDoubleFilterEval, default true) pushes a cheap
> conjunct below the projection ahead of an expensive conjunct written before
> it:
> {code}
> SELECT * FROM (SELECT id, f(id) AS f FROM range(10))
> WHERE f > 5 AND assert_true(id > 5) IS NULL
> {code}
> This fails regardless of the split config, and is in 4.2.0.
> Other filter pushdown rules (CombineFilters, PushPredicateThroughJoin)
> already keep throwable conditions in place.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
