pan3793 opened a new pull request, #59697: URL: https://github.com/apache/spark/pull/59697
### What changes were proposed in this pull request? This PR allows `CREATE [TEMPORARY] FUNCTION ... AS 'class' [USING JAR ...]` on the session catalog to register a DataSource V2 function, i.e. a class implementing `org.apache.spark.sql.connector.catalog.functions.UnboundFunction`, the same way as Hive UDFs. ```sql ADD JAR /path/to/my-udfs.jar; CREATE TEMPORARY FUNCTION strlen AS 'com.example.StrLen'; -- or persistent, with the jar loaded on first use CREATE FUNCTION strlen AS 'com.example.StrLen' USING JAR 'hdfs:///udfs/my-udfs.jar'; SELECT strlen(name) FROM t; ``` Persistent functions are stored in the external catalog in the same shape as Hive UDFs (real class name and resource URIs), with no new metadata format; the function kind is decided by the class when loaded. They behave like Hive UDFs: a missing class, an inaccessible jar, or a class that cannot be instantiated only fails when the function is invoked, not on CREATE, SHOW, DESCRIBE or listing. HMS calls are the same as for a Hive UDF. How other engines sharing the HMS react to such an entry: 1. Hive: an entry it cannot register is logged and skipped. 2. Flink: lists it, and fails only when it is called. 3. Impala: skips it with an error log. 4. Trino, Doris, StarRocks: do not read HMS function entries created by other engines. Flink and Impala store their own Java functions in this same shape. A class implementing both a Hive UDF interface and `UnboundFunction` runs as a V2 function in Spark and as a Hive UDF in other engines. ### Why are the changes needed? The V2 function API is the modern, codegen-friendly way to write Spark functions, but it is only usable through a custom `FunctionCatalog`. This PR lets session catalog users, including HMS, use V2 functions with the familiar `ADD JAR` / `CREATE FUNCTION` workflow, without implementing a catalog plugin. ### Does this PR introduce _any_ user-facing change? Yes. `CREATE [TEMPORARY] FUNCTION` accepts a class implementing `UnboundFunction`, and `DESCRIBE FUNCTION` on it shows the function's description. ### How was this patch tested? New `CreateV2FunctionSuite`, and a new test in `HiveUDFSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
