cloud-fan commented on code in PR #58450:
URL: https://github.com/apache/spark/pull/58450#discussion_r3991425946
##########
sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala:
##########
@@ -965,6 +956,36 @@ class SetPathSuite extends SharedSparkSession {
}
}
+ test("path-driven COUNT rewrite gate: DataFrame count(\"*\") and count(t.*)
reach the owner " +
+ "probe and expand through a shadowing temp count") {
+ // SQL `count(*)` is normalized to `count(1)` in AstBuilder, so it never
reaches the analyzer
+ // owner probe. A DataFrame `count("*")` keeps its UnresolvedStar and
does, as does count(t.*).
+ // A non-1 input distinguishes an incorrect `count(1)` rewrite from
correct star expansion to
+ // `count(a)` through the temp.
+ withPathEnabled {
+ sql("CREATE TEMPORARY FUNCTION count(x INT) RETURNS INT RETURN x + 100")
+ try {
+ val df = sql("SELECT * FROM VALUES (7) AS t(a)")
+
+ // Builtin-first: count is the builtin, so `count("*")` collapses to
`count(1)` and returns
+ // the row count (1), while `count(t.*)` hits the single-table-star
guard.
+ checkAnswer(df.select(functions.count("*")), Row(1))
+ intercept[AnalysisException] {
Review Comment:
**Non-blocking (P2):** This assertion accepts every `AnalysisException`, so
it remains green if the single-table-star guard stops raising
`INVALID_USAGE_OF_STAR_WITH_TABLE_IDENTIFIER_IN_COUNT` and an unrelated
analyzer path fails instead. Please capture the exception and check its
condition and message parameters with `checkError` (or the suite's established
structured-error helper) so the test proves the intended guard fired.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:
##########
@@ -404,6 +388,103 @@ class FunctionResolution(
}
}
+ /**
+ * Returns whether an unqualified function name reaches `system.builtin`
before any temp or
+ * persistent function in the effective SQL PATH. When a temp or persistent
function shadows the
+ * builtin, special-syntax handling that only applies to Spark's builtins
must not fire, since the
+ * name no longer refers to the builtin -- e.g. rejecting a direct star
(bare `*` or qualified
+ * `t.*`) in a routed SQL/JSON function or the `count(tbl.*)` guard.
Parser-built `count(*)` is
+ * normalized to `count(1)` in `AstBuilder` so it skips this probe, but a
DataFrame `count("*")`
+ * keeps its star and does reach the probe during analyzer normalization.
+ *
+ * Precondition: `functionName` must already be known to be a stock built-in
name (as
+ * `functionNameResolvesToBuiltin` ensures by checking
`FunctionRegistry.functionSet` first). This
+ * returns true as soon as the PATH reaches `system.builtin`, without
verifying that
+ * `system.builtin` actually defines a function of this name, so calling it
for a non-builtin name
+ * would wrongly report builtin ownership.
+ */
+ def unqualifiedFunctionResolvesToBuiltinBeforeAnyShadow(functionName:
String): Boolean = {
+ // Walk the PATH in order and stop at the first entry that owns the name.
The default order puts
+ // system.builtin first, so the common case returns on the first entry
with no catalog lookup;
+ // only a custom PATH that lists a persistent catalog ahead of
system.builtin reaches the probe
+ // below (one lookup per such preceding entry, recomputed on each call --
not cached).
+ sqlResolutionPathEntriesForAnalysis.foreach { pathEntry =>
+ val candidate = pathEntry :+ functionName
+ FunctionResolution.sessionNamespaceKind(candidate) match {
+ case
Some(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Builtin) =>
+ return true
+ case Some(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Temp) =>
+ // A visible temp scalar function shadows the builtin; a visible
temp *table* function
+ // makes scalar resolution terminal at this PATH entry
(NOT_A_SCALAR_FUNCTION). Either way
+ // the name never reaches system.builtin, mirroring
`resolveFunctionCandidate`.
+ val ident = FunctionIdentifier(functionName)
+ if (v1SessionCatalog.isTemporaryScalarFunctionVisible(ident) ||
+ v1SessionCatalog.isTemporaryTableFunctionVisible(ident)) {
+ return false
+ }
+ case None =>
+ if (persistentFunctionExists(candidate)) {
+ return false
+ }
+ }
+ }
+ false
+ }
+
+ /**
+ * Returns true when a function reference resolves to the system built-in
with the requested name.
+ * This mirrors [[resolveFunction]] for special parser/analyzer rewrites
that must run only for
+ * Spark's built-ins. In particular, two-part `builtin.name` is not always a
system built-in:
+ * with `spark.sql.legacy.persistentCatalogFirst=true`, an existing
persistent
+ * `current_catalog.builtin.name` takes precedence.
+ */
+ def functionNameResolvesToBuiltin(nameParts: Seq[String], expectedName:
String): Boolean = {
+ if (!FunctionRegistry.functionSet.contains(
+ FunctionRegistry.builtinFunctionIdentifier(expectedName)) ||
+ !FunctionResolution.isUnqualifiedOrBuiltinFunctionName(nameParts,
expectedName)) {
+ return false
+ }
+ nameParts.length match {
+ case 1 =>
+ unqualifiedFunctionResolvesToBuiltinBeforeAnyShadow(nameParts.head)
+ case 2 =>
+ conf.prioritizeSystemCatalog || !persistentFunctionExists(nameParts)
+ case 3 =>
+ true
+ case _ =>
+ false
+ }
+ }
+
+ // All routed SQL/JSON functions (JSON_ARRAY, JSON_VALUE, JSON_QUERY,
JSON_EXISTS) forbid a bare
Review Comment:
**Nit (P3):** The shared guard rejects any direct `Star`, including
qualified `t.*`, but these lines describe only a bare `*`. Please call this a
direct star argument (bare `*` or qualified `t.*`) here and in
`JsonArraySuite`'s `Only a bare * element is rejected` comment, while retaining
the distinction from a star nested inside another expression.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala:
##########
@@ -950,9 +950,33 @@ object FunctionRegistry {
expression[SchemaOfJson]("schema_of_json"),
expression[LengthOfJsonArray]("json_array_length"),
expression[JsonObjectKeys]("json_object_keys"),
- expression[JsonTypeof]("json_typeof")
+ expression[JsonTypeof]("json_typeof"),
+ // Built-in forms of the SQL:2016 JSON constructor and path functions,
resolved for plain calls
Review Comment:
**Nit (P3):** This says plain calls without SQL/JSON clauses resolve through
these builders, but `AstBuilder` still directly constructs clause-free
`JSON_ARRAY`/`JSON_QUERY` in the nested or implicit-JSON cases. Please qualify
this as the eligible flat clause-free forms and point to the intentional
SPARK-59243 exception, so the registration guidance matches the accepted
routing boundary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
