haiyangsun-db opened a new pull request, #58713:
URL: https://github.com/apache/spark/pull/58713
### What changes were proposed in this pull request?
This PR adds the session initialization metadata needed by the
language-neutral external UDF
framework:
* Extend worker specifications with declarations for static and named
dynamic configuration,
authorized environment variables, and required resource directories.
* Build complete `Init` messages from `ExternalUserDefinedFunction`,
including logical input
metadata, named arguments, invocation identity, payload format, and
evaluation type.
* Add a versioned PySpark UDF payload and a bridge from scalar Arrow
`PythonUDF` expressions.
* Reuse the Arrow Python runner's configuration declarations when
constructing Python worker
specifications.
The existing Python UDF execution path is not switched to unified execution
by this PR. Python
worker integration and broadcast-variable support are tracked separately by
[SPARK-59368](https://issues.apache.org/jira/browse/SPARK-59368) and
[SPARK-59366](https://issues.apache.org/jira/browse/SPARK-59366),
respectively.
### Why are the changes needed?
The external UDF framework can describe and launch workers, but it does not
yet have a generic
contract for the per-session engine context or invocation metadata a worker
needs. Without that
contract, physical execution would need language-specific knowledge, and
Catalyst child rewrites
could leave serialized input metadata stale.
These changes keep worker requirements declarative, forward only
engine-authorized context, and
rebuild logical input metadata from the current expression children.
### Does this PR introduce _any_ user-facing change?
No. This extends unreleased external UDF infrastructure and does not change
the execution path of
existing Python UDFs.
### How was this patch tested?
Added `PythonExternalUserDefinedFunctionSuite` coverage for context
selection, required and
optional dynamic configuration, conflicting targets, named resource
directories, rewritten and
named input metadata, Python worker configuration declarations, unsupported
evaluation types, and
versioned payload decoding.
The protocol module and SQL tests were compiled, then the focused suites
were run with:
```bash
build/sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \
udf-worker-proto/compile \
sql/Test/compile \
'sql/testOnly
org.apache.spark.sql.execution.externalUDF.PythonExternalUserDefinedFunctionSuite
org.apache.spark.sql.execution.externalUDF.PythonUDFWorkerSpecificationSuite
org.apache.spark.sql.execution.externalUDF.PlanExternalUDFsSuite'
```
All 39 executed tests passed. One environment-gated Python worker test was
canceled because
PySpark was unavailable in the local Python environment.
### Was this patch authored or co-authored using generative AI tooling?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
