Chao Sun created SPARK-58310:
--------------------------------
Summary: Avoid runtime Bloom-filter subqueries containing Python
UDFs
Key: SPARK-58310
URL: https://issues.apache.org/jira/browse/SPARK-58310
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.3.0
Reporter: Chao Sun
When runtime Bloom filtering and adaptive query execution are enabled,
InjectRuntimeFilter can build a Bloom-filter scalar subquery from a
creation-side logical plan that contains a Python UDF. Runtime filter
subqueries are introduced after subquery optimization, so the Python UDF in the
newly constructed subquery is not extracted into a Python evaluation operator.
Physical planning then reaches PythonUDF.doGenCode while executing the
BloomFilterAggregate, causing an unevaluable-expression/code-generation failure.
A nested query using a Python-normalized dimension, DISTINCT, an explicitly
broadcast join, and a scalar aggregate reproduces the problem against Apache
Spark master. Both standard scalar Python UDFs and Arrow-backed scalar pandas
UDFs are affected.
Skip only the runtime Bloom filter whose creation-side plan contains a Python
UDF. Continue to inject safe Bloom filters when a Python UDF appears only on
the application side. Add a regression that checks adaptive execution, exact
query results, a Parquet round trip, standard and Arrow UDFs, and continued
safe Bloom-filter injection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
