[
https://issues.apache.org/jira/browse/SPARK-61005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61005:
-----------------------------------
Labels: pull-request-available (was: )
> Create the UDFs of udf of the Python client as PySpark does
> -----------------------------------------------------------
>
> Key: SPARK-61005
> URL: https://issues.apache.org/jira/browse/SPARK-61005
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{udf(f, returnType)}} without {{useArrow}}
> ({{_create_py_udf}}) makes an Arrow-optimized UDF ({{SQL_ARROW_BATCHED_UDF}})
> when the active session enables
> {{spark.sql.execution.pythonUDF.arrow.enabled}}, which the 4.3.0 server does
> by default, and infers the eval type of a function with type hints
> ({{infer_eval_type_for_udf}}): a function of {{pd.Series}} makes a pandas UDF
> ({{SQL_SCALAR_PANDAS_UDF}}), and one of {{pa.Array}} an Arrow UDF
> ({{SQL_SCALAR_ARROW_UDF}}). It returns the function wrapper of the UDF
> ({{UserDefinedFunction._wrapped}}), with the {{__name__}} and {{__doc__}} of
> {{f}} and the {{func}}, {{returnType}}, {{evalType}}, {{deterministic}},
> {{asNondeterministic}} and {{_unwrapped}} of the UDF.
> The Rust-backed Python client always makes a {{SQL_BATCHED_UDF}}, so the
> results differ from those of PySpark: for {{udf(lambda x: str(x), "int")}},
> PySpark 4.3.0 returns {{0}} for {{0}}, as Arrow converts the string, and the
> client {{None}}. It also returns the {{UserDefinedFunction}} rather than its
> wrapper, which has no {{__name__}} or {{__doc__}}. Create UDFs as PySpark
> does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
