L. C. Hsieh created SPARK-60633:
-----------------------------------
Summary: Do not push a throwable filter condition through a unary
operator ahead of the conditions written before it
Key: SPARK-60633
URL: https://issues.apache.org/jira/browse/SPARK-60633
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
The generic UnaryNode case of PushPredicateThroughNonJoin (pushDownPredicate)
pushes every deterministic conjunct whose references are available below the
operator, and keeps the rest above. It does not check Expression.throwable, so
a throwable conjunct (raise_error / assert_true) can be pushed below the
operator ahead of a conjunct written before it that stays above. The throwable
conjunct then runs on rows the AND short-circuit would have kept from it, and
the query fails.
This shows up with Python UDFs: after ExtractPythonUDFs, SparkOptimizer reruns
PushPredicateThroughNonJoin, and BatchEvalPython / ArrowEvalPython are in
canPushThrough. For example:
{code:python}
from pyspark.sql.functions import col, udf, assert_true
@udf("long")
def f(x):
return x
spark.range(0, 10, 1, 1).select("id", f("id").alias("f")) \
.where((col("f") > 5) & assert_true(col("id") > 5).isNull()).collect()
{code}
fails with USER_RAISED_EXCEPTION, with
spark.sql.optimizer.avoidDoubleFilterEval either true or false and for both
regular and pandas UDFs, because `assert_true(id > 5) IS NULL` is pushed alone
below the ArrowEvalPython node:
{code}
ArrowEvalPython [f(id)], [pythonUDF0]
+- Filter if ((id > 5)) true else isnull(raise_error(...))
+- Range (0, 10, step=1, splits=1)
{code}
The same applies to the other operators in canPushThrough (Generate, Expand,
Sort, Repartition, ...) whenever an earlier conjunct cannot be pushed through
them.
SPARK-60602 fixes the same problem for the Filter-over-Project case.
CombineFilters and PushPredicateThroughJoin already keep throwable conditions
in place.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
