Holden Karau created SPARK-55206:
------------------------------------
Summary: Support basic single parameter addition Python UDF
transpilation
Key: SPARK-55206
URL: https://issues.apache.org/jira/browse/SPARK-55206
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.3
Reporter: Holden Karau
For the sake of having code land in testable/reviewable chunks I think the
first chunk we should do should be the bare bones for simple numerical PyUDF
transpilation. To accomplish that we'll need to include the following:
* Parameter for enabing transpilation (clearly marked as experimental)
* AST extraction
* AST transpilation
* UDF node extended to support both transpiled and "original" UDF code
(possible with an explanation string for why no transpilation)
* UDF node printout in the query plan should be clear if transpiled or not
* Property-based tests for transpiled UDFS ensuring correctness
* Type promotion equivalence (does not need to be complete given numeric only)
My theory here is that by putting all of this together in one first PR, we both
establish a solid framework for future work and ensure the framework we land is
actually usable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
