Thomas Newton created SPARK-61451:
-------------------------------------
Summary: Inconsistent default for whether arrow is used in UDFs
Key: SPARK-61451
URL: https://issues.apache.org/jira/browse/SPARK-61451
Project: Spark
Issue Type: Improvement
Components: UDF
Affects Versions: 4.2.1, 4.3.0, 4.2.0
Reporter: Thomas Newton
Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to
`true`.
However when creating a python UDF at import time. For example if the UDF is
defined using a decorator
```
from pyspark.sql.functions import udf
@udf(returnType=...)
def some_udf()
```
Whether this UDF uses arrow or not, now depends on whether the spark session
was setup prior to importing this file, because of the loigic here
[https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]
Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs
dissallow a python list being converted to a struct. For us this causes
different behaviour during tests vs in real deployments, because tests create
the spark session after import time, using `pytest` fixtures, but in
deployments the spark session is created before import time by spark-submit.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]