Thomas Newton created SPARK-61451:
-------------------------------------

             Summary: Inconsistent default for whether arrow is used in UDFs
                 Key: SPARK-61451
                 URL: https://issues.apache.org/jira/browse/SPARK-61451
             Project: Spark
          Issue Type: Improvement
          Components: UDF
    Affects Versions: 4.2.1, 4.3.0, 4.2.0
            Reporter: Thomas Newton


Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to 
`true`.

However when creating a python UDF at import time. For example if the UDF is 
defined using a decorator

```

from pyspark.sql.functions import udf

 

@udf(returnType=...)
def some_udf()

```

Whether this UDF uses arrow or not, now depends on whether the spark session 
was setup prior to importing this file, because of the loigic here 
[https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]

Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs 
dissallow a python list being converted to a struct. For us this causes 
different behaviour during tests vs in real deployments, because tests create 
the spark session after import time, using `pytest` fixtures, but in 
deployments the spark session is created before import time by spark-submit. 

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to