sunchao commented on code in PR #59697:
URL: https://github.com/apache/spark/pull/59697#discussion_r4349587567
##########
docs/sql-ref-syntax-ddl-create-function.md:
##########
@@ -82,6 +82,8 @@ CREATE [ OR REPLACE ] [ TEMPORARY ] FUNCTION [ IF NOT EXISTS ]
Specifies the name of the class that provides the implementation for
function to be created.
The implementing class should extend one of the base classes as follows:
+ * Should implement `UnboundFunction` in
`org.apache.spark.sql.connector.catalog.functions`
+ package.
Review Comment:
[P2] Publish the runtime-JAR restriction with the new function support
This reference now advertises `UnboundFunction` alongside the usual `USING
JAR` workflow, but the accepted magic-method limitation appears only in the PR
description. When query compilation runs with a context classloader that cannot
see the runtime-added jar, such as a later spark-shell line or an existing
driver-pool thread, generated code cannot resolve the user class. Whole-stage
codegen can fall back, while an ordinary `ORDER BY f(c) LIMIT 10` can fail
because its generated ordering has no fallback. The source paths support this
limitation, and the current persistent-JAR test deliberately uses `magic =
false`.
Please describe this restriction in the shipped reference (or a linked
V2-function section), including the startup `--jars` / `spark.jars` and
produceResult-only alternatives. The same section could state the public
no-argument constructor requirement. The underlying classloader fix can remain
a separate follow-up. My local JAR probe encountered a deserialization failure
before top-K, so this comment relies on the source trace and the already
acknowledged limitation.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/SessionCatalog.scala:
##########
@@ -2167,12 +2212,56 @@ class SessionCatalog(
if (funcDefinition.isUserDefinedFunction) {
UserDefinedFunction.fromCatalogFunction(funcDefinition,
parser).toExpressionInfo
} else {
- makeExprInfoForHiveFunction(funcDefinition)
+ // Registration builds the function, which loads its class.
+ makeExprInfoForClassFunction(funcDefinition, resourcesLoaded = true)
}
}
registry.registerFunction(identToRegister, resolvedInfo, functionBuilder)
}
+ /**
+ * Builds the [[ExpressionInfo]] of a class-based function (`CREATE FUNCTION
... AS 'class'`).
+ * A DataSource V2 function reports its own description, which requires
instantiating it.
+ * Otherwise, or if the class cannot be loaded or instantiated, use the
class-name-only info, so
+ * that like a Hive UDF, a broken function only fails when invoked. Before
the function's `USING`
+ * resources are loaded (`resourcesLoaded = false`), its class is not
touched, as initializing it
+ * may require those resources.
+ */
+ private def makeExprInfoForClassFunction(
+ func: CatalogFunction,
+ resourcesLoaded: Boolean): ExpressionInfo = {
+ if (!resourcesLoaded && func.resources.nonEmpty) {
+ return makeExprInfoForHiveFunction(func)
+ }
+ try {
+ if (Utils.classIsLoadableAndAssignableFrom(func.className,
classOf[UnboundFunction])) {
+ val clazz = Utils.classForName(func.className)
+ makeExprInfoForV2Function(func, newV2FunctionInstance(clazz,
func.identifier))
Review Comment:
[P3, nonblocking] Reuse the registration instance for its description
`makeFunctionBuilder` has already constructed and retained the
`UnboundFunction`, but registration calls this helper with `info = None` and
constructs a second object solely for `description()`. A counting-constructor
probe against the exact-head CI assembly observed two constructions for
temporary CREATE. Persistent CREATE performed none; each uncached DESCRIBE
added one, and first invocation added three more before the metadata became
cached.
Could the registration path obtain its builder and `ExpressionInfo` from the
same V2 instance and pass that info through the existing parameter? That
removes demonstrable duplicate constructor work while preserving the
metadata-only fallback path. This is an optional improvement; no latency
regression or resource leak is claimed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
