cloud-fan commented on code in PR #58450:
URL: https://github.com/apache/spark/pull/58450#discussion_r3936113072
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -4267,38 +4299,42 @@ class AstBuilder extends DataTypeAstBuilder
*/
override def visitJsonArray(ctx: JsonArrayContext): Expression =
withOrigin(ctx) {
val arrayValues = ctx.values.asScala.map(v => expression(v.value)).toSeq
- // Freeze the FORMAT JSON decisions here, from the lexical argument, so a
later
- // analyzer/optimizer rewrite that wraps or swaps the child cannot change
them (see
- // [[ImplicitlyFormattedAsJson]]). For each element:
- // - `formatJson`: whether it is already-JSON text spliced raw. True when
it carries an
- // explicit `FORMAT JSON` clause, or is a (lexically) nested JSON
constructor -- seen through
- // a value-preserving `COLLATE` via `JsonArray.isImplicitlyJson`.
- // - `needsValidation`: whether its raw text is arbitrary user input to
JSON-validate at eval.
- // True only for an explicit `FORMAT JSON` on something that is NOT a
JSON constructor; a
- // nested constructor emits well-formed JSON by construction and is
trusted.
- val formatArgs = ctx.values.asScala.zip(arrayValues).map { case (v, expr)
=>
- val explicit = v.FORMAT() != null
- val implicitlyJson = JsonArray.isImplicitlyJson(expr)
- (explicit || implicitlyJson, explicit && !implicitlyJson)
- }.toSeq
- val formatJson = formatArgs.map(_._1)
- val needsValidation = formatArgs.map(_._2)
- // Default RETURNING type is STRING; the result is JSON text. A
CHAR/VARCHAR RETURNING is
- // normalized to STRING unconditionally: JSON_ARRAY serializes the
fragment itself and never
- // advertises a CHAR/VARCHAR length it does not enforce. The
CharVarcharUtils helpers cannot be
- // used here -- they honor spark.sql.preserveCharVarcharTypeInfo and would
leave a VARCHAR(n)
- // length in the output type when that flag is set. A non-string RETURNING
is left intact for
- // checkInputDataTypes to fail.
- val returning = Option(ctx.returning).map(typedVisit[DataType]).map {
- case c: CharType => c.toStringType
- case v: VarcharType => v.toStringType
- case other => other
- }.getOrElse(StringType)
- // Default ON NULL behavior is ABSENT ON NULL (drop NULL elements).
- val nullBehavior = Option(ctx.nullBehavior)
- .map(buildJsonConstructorNullBehavior)
- .getOrElse(JsonConstructorNullBehavior.Absent)
- JsonArray(arrayValues, formatJson, needsValidation, nullBehavior,
returning)
+ val hasExplicitFormat = ctx.values.asScala.exists(_.FORMAT() != null)
+ val hasImplicitJson = arrayValues.exists(JsonArray.isImplicitlyJson)
+ if (!hasExplicitFormat && ctx.nullBehavior == null && ctx.returning ==
null &&
+ !isTopLevelJsonArrayElement(ctx) && !hasImplicitJson) {
Review Comment:
**Blocking (P1):** This argument-shape exception bypasses the PR's core
shadowing behavior. With a temporary `json_array` first on PATH,
`json_array(json_array(1))` takes this direct branch and executes the Spark
builtin; the qualified routed spelling reaches the builder but quotes the
nested document because the lexical FORMAT JSON intent was lost. Please route
every clause-free call through normal ownership resolution and carry
per-argument implicit-JSON metadata so only the selected builtin applies the
original splice flags.
**Recommended change:** Carry lexical implicit-FORMAT-JSON intent through
the unresolved routine-resolution boundary instead of using it to suppress
resolution.
**Why this works:** Represent the parser's per-argument JSON-format flags on
the unresolved call or an equivalent resolution-safe carrier; resolve the owner
normally, then let only JsonArrayExpressionBuilder consume those flags when the
Spark builtin wins.
**Scope:** AstBuilder JSON_ARRAY/JSON_QUERY routing, unresolved function
metadata or equivalent carrier, JSON expression builders, and nested
PATH/qualification regressions.
**Compatibility:** Preserve existing clause-bearing direct syntax, nested
FORMAT JSON splicing, collation behavior, and ordinary arguments to user
routines while making clause-free ownership independent of argument shape.
**Risks:** Generic routine resolution must not expose builtin-only metadata
to temporary or persistent functions. Analyzer rewrites and casts must not
erase or re-derive the frozen lexical splice decision.
**Constraints:** All clause-free calls must select their owner through the
same PATH and qualification rules. A selected shadow routine must receive
normal arguments without builtin FORMAT JSON handling. A selected builtin must
preserve existing nested JSON values and output collation.
**Success:** Under a shadow-first PATH, nested clause-free calls select the
shadow just like flat calls; explicit builtin qualification selects Spark's
implementation without changing nested JSON from spliced to quoted.
##########
sql/core/src/test/scala/org/apache/spark/sql/SetPathSuite.scala:
##########
@@ -937,9 +937,9 @@ class SetPathSuite extends SharedSparkSession {
test("path-driven COUNT(*) rewrite gate: temp count shadowing builtin under
SET PATH " +
"(session-first) suppresses the * -> 1 rewrite") {
- // `Analyzer.matchesFunctionName` consults
- // `FunctionResolution.isSessionBeforeBuiltinInPath` to decide whether
COUNT(*) is the
- // builtin (eligible for the COUNT(*) -> COUNT(1) shortcut) or a
user-defined override.
+ // `Analyzer.matchesFunctionName` consults
`FunctionResolution.functionNameResolvesToBuiltin`
Review Comment:
**Non-blocking (P2):** These SQL-string cases do not exercise the owner
probe: `AstBuilder` has already converted unqualified `count(*)` to `count(1)`.
The qualified case also uses `VALUES (1)`, so correct expansion to the column
and an incorrect rewrite to literal 1 both return 101. Please add a DataFrame
`count("*")` case over a non-1 (ideally nullable) value and a shadow-owned
`count(t.*)` case, covering both analyzer modes and `persistentCatalogFirst` so
the assertions distinguish every owner-dependent branch.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala:
##########
@@ -2105,21 +2105,11 @@ class Analyzer(
* This is used for special syntax transformations (e.g., COUNT(*) ->
COUNT(1)) that
* should only apply to builtin functions, not to user-defined functions.
*
- * When the effective SQL PATH puts `system.session` before
`system.builtin`, temp
- * functions shadow builtins, so an unqualified name that matches a temp
function
- * should NOT be treated as builtin.
+ * Mirrors function resolution precedence, including SQL PATH shadowing
for unqualified names
+ * and `spark.sql.legacy.persistentCatalogFirst` for two-part
`builtin.name` references.
*/
- private def matchesFunctionName(nameParts: Seq[String], expectedName:
String): Boolean = {
- if (!FunctionResolution.isUnqualifiedOrBuiltinFunctionName(nameParts,
expectedName)) {
- return false
- }
- if (nameParts.size == 1 &&
functionResolution.isSessionBeforeBuiltinInPath) {
- val v1Catalog = catalogManager.v1SessionCatalog
- !v1Catalog.isTemporaryFunction(FunctionIdentifier(nameParts.head))
- } else {
- true
- }
- }
+ private def matchesFunctionName(nameParts: Seq[String], expectedName:
String): Boolean =
+ functionResolution.functionNameResolvesToBuiltin(nameParts, expectedName)
Review Comment:
**Non-blocking (P2):** `matchesFunctionName` can now walk persistent PATH
entries via external `FunctionCatalog.functionExists`, but this guard probes
before the cheap star-shape check and the fallback count guard can probe the
same node again when a shadow owns `count`. Please mirror
`FunctionResolverUtils`: check local argument shape first, lazy-compute the
count owner once per unresolved function, and reuse it for normalization and
the table-star guard.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala:
##########
@@ -466,18 +492,272 @@ class JsonArraySuite extends QueryTest with
SharedSparkSession {
}
}
+ test("plain call goes through routine resolution and can be shadowed via SET
PATH") {
+ // `withUserDefinedFunction` is unusable here: its cleanup asserts the
name no longer resolves,
+ // but `json_array` is now a registered built-in, so drop the temporary
routine explicitly.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_array(a INT, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_value(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_query(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_exists(a STRING, b STRING) RETURNS
BOOLEAN " +
+ "RETURN false")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine
(ahead of
+ // system.builtin on the path) shadows the built-in constructor.
+ checkAnswer(sql("SELECT json_array(1, 'x')"), Row("shadowed"))
+ checkAnswer(sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a,
b)"), Row("shadowed"))
+ // The clause-bearing form is not a function call, so it stays the
built-in constructor.
+ checkAnswer(sql("SELECT json_array('x' NULL ON NULL)"),
Row("""["x"]"""))
+ // Nested JSON-producing children stay on the direct-construction
path, so they are not
+ // shadowed. This preserves JSON_ARRAY's parse-time splice decisions.
+ checkAnswer(sql("SELECT json_array(json_array(1))"), Row("[[1]]"))
+ checkAnswer(
+ sql("""SELECT json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""[{"x":1}]"""))
+ // Plain scalar and predicate children are still ordinary function
calls. Use an explicit
+ // outer NULL clause to keep the parent on the direct path while the
children are shadowed.
+ checkAnswer(
+ sql("""SELECT json_array(json_value('{"a":"x"}', '$.a') NULL ON
NULL)"""),
+ Row("""["shadowed"]"""))
+ checkAnswer(
+ sql("""SELECT json_array(json_exists('{"a":1}', '$.a') NULL ON
NULL)"""),
+ Row("[false]"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_array")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_value")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_query")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_exists")
+ }
+ }
+ }
+
+ test("qualified plain JSON_ARRAY resolves to the built-in constructor") {
+ checkAnswer(sql("SELECT builtin.json_array(1, 'x')"), Row("""[1,"x"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(1, 'x')"),
Row("""[1,"x"]"""))
+ }
+
+ test("a nested JSON constructor through a routed JSON_ARRAY call is quoted,
not spliced") {
+ // A routed (plain or qualified) call carries no lexical FORMAT JSON, so a
nested JSON
+ // constructor argument is treated as a plain value and quoted, unlike the
JSON_ARRAY(...)
+ // grammar which splices it (see the unqualified
`json_array(json_array(1))` -> `[[1]]` cases
+ // above). A nested constructor reaches the routed builder only via a
qualified outer call:
+ // an unqualified nested constructor stays on the direct grammar path.
Splicing through a
+ // routed call is left as a follow-up.
+ checkAnswer(sql("SELECT builtin.json_array(json_array(1))"),
Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(json_array(1))"),
Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT builtin.json_array(json_array(1), 2)"),
Row("""["[1]",2]"""))
+ checkAnswer(
+ sql("""SELECT builtin.json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""["{\"x\":1}"]"""))
+ }
+
+ test("invalid: a bare star argument in plain JSON_ARRAY is not expanded") {
+ Seq("json_array", "builtin.json_array",
"system.builtin.json_array").foreach { func =>
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT $func(*) FROM VALUES (1, 'x') AS t(a, b)").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for
$func(*)")
+ }
+ }
+
+ test("invalid: a bare star argument in clause-bearing JSON_ARRAY is not
expanded") {
+ val e = intercept[AnalysisException] {
+ sql("SELECT json_array(* NULL ON NULL) FROM VALUES (1, 'x') AS t(a,
b)").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX")
+ }
+
+ test("JSON_ARRAY expands a star nested in a sibling constructor (array(*))")
{
+ // Only a bare `*` element is rejected. A star nested in `array(...)`
belongs to that call and
+ // is expanded there, exactly as `array(array(*))` would, then JSON_ARRAY
wraps the result.
+ checkAnswer(
+ sql("SELECT json_array(array(*)) FROM VALUES (1, 2) AS t(a, b)"),
+ Row("[[1,2]]"))
+ // Clause-bearing form (a direct-construction JsonArray node) behaves the
same.
+ checkAnswer(
+ sql("SELECT json_array(array(*) NULL ON NULL) FROM VALUES (1, 2) AS t(a,
b)"),
+ Row("[[1,2]]"))
+ // Alongside count(*): the array's star expands, count(*) is rewritten,
neither is rejected.
+ checkAnswer(
+ sql("SELECT json_array(count(*), array(max(a))) FROM VALUES (1), (2) AS
t(a)"),
+ Row("[2,[2]]"))
+ }
+
+ test("single-pass: JSON_ARRAY expands a star nested in array(*)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(array(*)) FROM VALUES (1, 2) AS t(a, b)",
+ "SELECT json_array(array(*) NULL ON NULL) FROM VALUES (1, 2) AS t(a,
b)"
+ ).foreach { query =>
+ // Analyze only: the single-pass analyzer cannot execute every
operator, so assert the
+ // nested star is neither rejected nor left unexpanded rather than
running it.
+ val analyzed = sql(query).queryExecution.analyzed
+ assert(analyzed.resolved, s"for $query")
+
assert(!analyzed.exists(_.expressions.exists(_.exists(_.isInstanceOf[Star]))),
+ s"star should not survive analysis for $query")
+ }
+ }
+ }
+
+ test("single-pass rejects a bare star in plain and clause-bearing JSON_ARRAY
built-ins") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)",
+ "SELECT json_array(* NULL ON NULL) FROM VALUES (1, 'x') AS t(a, b)"
+ ).foreach { query =>
+ val e = intercept[AnalysisException] {
+ spark.sql(query).queryExecution.analyzed
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for
$query")
+ }
+ }
+ }
+
+ test("JSON_ARRAY accepts count(*): it is normalized to count(1), not
star-expanded") {
+ // count(*) is rewritten to count(1) rather than star-expanded, so it
stays a valid aggregate
+ // argument to the JSON_ARRAY built-in. The star pre-check must not reject
the nested star.
+ checkAnswer(
+ sql("SELECT json_array(count(*)) FROM VALUES (1), (2), (3) AS t(a)"),
+ Row("[3]"))
+ // Alongside another aggregate argument.
+ checkAnswer(
+ sql("SELECT json_array(count(*), max(a)) FROM VALUES (1), (2), (3) AS
t(a)"),
+ Row("[3,3]"))
+ // The clause-bearing form (a direct-construction JsonArray node) accepts
it too.
+ checkAnswer(
+ sql("SELECT json_array(count(*) NULL ON NULL) FROM VALUES (1), (2), (3)
AS t(a)"),
+ Row("[3]"))
+ // Qualified built-in references resolve to the same built-in and behave
the same.
+ checkAnswer(
+ sql("SELECT builtin.json_array(count(*)) FROM VALUES (1), (2), (3) AS
t(a)"),
+ Row("[3]"))
+ }
+
+ test("single-pass: JSON_ARRAY accepts count(*)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(count(*)) FROM VALUES (1), (2), (3) AS t(a)",
+ "SELECT json_array(count(*) NULL ON NULL) FROM VALUES (1), (2), (3) AS
t(a)"
+ ).foreach { query =>
+ // Analyze only: the single-pass analyzer cannot execute every
operator, so we assert the
+ // query resolves without INVALID_USAGE_OF_STAR_OR_REGEX rather than
running it.
+ val analyzed = sql(query).queryExecution.analyzed
+ assert(analyzed.resolved, s"for $query")
+
assert(!analyzed.exists(_.expressions.exists(_.exists(_.isInstanceOf[Star]))),
+ s"star should not survive analysis for $query")
+ }
+ }
+ }
+
+ test("invalid: a bare star next to count(*) is still rejected in
JSON_ARRAY") {
+ // count(*) is excluded from the star check, but a bare `*` element still
would be expanded and
+ // must be rejected, even when it sits next to a count(*).
+ Seq(false, true).foreach { singlePass =>
+ withSQLConf(
+ SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key ->
singlePass.toString) {
+ val e = intercept[AnalysisException] {
+ sql("SELECT json_array(count(*), *) FROM VALUES (1, 'x') AS t(a, b)")
+ .queryExecution.analyzed
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX",
s"singlePass=$singlePass")
+ }
+ }
+ }
+
+ test("plain JSON_ARRAY with star can be shadowed by a persistent function in
PATH") {
+ withSQLConf(SQLConf.PATH_ENABLED.key -> "true") {
+ withDatabase("path_json_array") {
+ sql("CREATE DATABASE path_json_array")
+ sql("CREATE FUNCTION path_json_array.json_array(a INT, b STRING)
RETURNS STRING " +
+ "RETURN 'persistent'")
+ try {
+ sql("SET PATH = spark_catalog.path_json_array, system.builtin")
+ checkAnswer(
+ sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a, b)"),
+ Row("persistent"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP FUNCTION IF EXISTS path_json_array.json_array")
+ }
+ }
+ }
+ }
+
+ test("two-part builtin JSON_ARRAY respects persistentCatalogFirst before
rejecting star") {
+ withDatabase("builtin") {
+ sql("CREATE DATABASE builtin")
+ sql("CREATE FUNCTION builtin.json_array(a INT, b STRING) RETURNS STRING
" +
+ "RETURN 'persistent'")
+ try {
+ val query = "SELECT builtin.json_array(*) FROM VALUES (1, 'x') AS t(a,
b)"
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "false") {
+ val e = intercept[AnalysisException] {
+ sql(query).collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX")
+ }
+ withSQLConf(SQLConf.PERSISTENT_CATALOG_FIRST.key -> "true") {
+ checkAnswer(sql(query), Row("persistent"))
+ }
+ } finally {
+ sql("DROP FUNCTION IF EXISTS builtin.json_array")
+ }
+ }
+ }
+
+ test("view-context shadow probe expands identifiers through the view's
frozen catalog") {
+ // The shadow probe must mirror `resolveFunctionCandidate`'s identifier
expansion. A permanent
+ // view freezes its creation catalog (spark_catalog). When the view is
read while a different
+ // catalog is current, the star pre-check for `builtin.json_array(*)` must
resolve `builtin`
+ // under the view's frozen catalog -- not the reader's current catalog --
so the persistent
+ // `spark_catalog.builtin.json_array` shadows the built-in and the star is
NOT rejected.
+ // Probing the reader's current catalog (the pre-fix behavior) misses the
persistent function
+ // and wrongly rejects the star.
+ withSQLConf(
+ "spark.sql.catalog.other_cat" -> classOf[InMemoryCatalog].getName,
+ SQLConf.PERSISTENT_CATALOG_FIRST.key -> "true") {
+ withDatabase("builtin") {
+ sql("CREATE DATABASE builtin")
+ sql("CREATE FUNCTION builtin.json_array(a INT, b STRING) RETURNS
STRING " +
+ "RETURN 'persistent'")
+ try {
+ sql("SET CATALOG spark_catalog")
+ sql("CREATE VIEW spark_catalog.default.json_array_shadow_view AS " +
+ "SELECT builtin.json_array(*) AS r FROM VALUES (1, 'x') AS t(a,
b)")
+ // Read the view while a different catalog is current: resolution
must still find the
+ // persistent function under the view's frozen spark_catalog.
+ sql("SET CATALOG other_cat")
+ checkAnswer(
+ sql("SELECT r FROM spark_catalog.default.json_array_shadow_view"),
+ Row("persistent"))
+ } finally {
+ sql("SET CATALOG spark_catalog")
+ sql("DROP VIEW IF EXISTS
spark_catalog.default.json_array_shadow_view")
+ sql("DROP FUNCTION IF EXISTS builtin.json_array")
+ }
+ }
+ }
+ }
+
test("default collation recurses into a nested JSON_ARRAY value") {
- // The rule casts each DefaultStringProducingExpression, recursing through
a nested constructor
- // (the flat cases above only cover a top-level constructor). This CTAS
runs the default
- // analyzer (single-pass included). Confirm the schema collation and that
raw splicing still
- // produces well-formed nested JSON at runtime.
+ // Col a (parser-built nested, direct grammar path) and col b (flat routed
built-in) both
+ // recolor to the table default collation.
Review Comment:
**Nit (P3):** `recolor` is not the operation being tested here. Both columns
adopt (or are collated with) the table default `UTF8_LCASE` collation; please
use that terminology.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonQuerySuite.scala:
##########
@@ -34,6 +34,36 @@ class JsonQuerySuite extends QueryTest with
SharedSparkSession {
private val doc =
"""{"id":7,"name":"Ada","tags":["x","y"],"addr":{"city":"NYC"},"score":null}"""
+ test("plain call goes through routine resolution and can be shadowed via SET
PATH") {
+ // `json_query` is now a registered built-in, so
`withUserDefinedFunction`'s cleanup assertion
+ // does not fit; drop the temporary routine explicitly instead.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_query(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine
shadows the
+ // built-in function.
+ checkAnswer(sql(s"SELECT json_query('$doc', '$$.addr')"),
Row("shadowed"))
+ checkAnswer(sql(s"SELECT json_query(*, '$$.addr') FROM VALUES ('$doc')
AS t(j)"),
+ Row("shadowed"))
+ // The clause-bearing form is not a function call, so it stays the
built-in function.
Review Comment:
**Nit (P3):** The clause-bearing form is still a `JSON_QUERY` function call.
The relevant distinction is that it is constructed directly by the dedicated
grammar branch instead of being routed through ordinary routine resolution.
Please describe that boundary rather than saying it is not a function call.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala:
##########
@@ -1651,6 +1651,132 @@ object JsonArray {
}
}
+// Built-in forms for the plain SQL/JSON constructor and path-function calls
that `AstBuilder`
+// routes through function resolution. Each rebuilds its expression with the
standard clause
+// defaults when unshadowed.
+
+@ExpressionDescription(
+ usage = "_FUNC_([expr[, ...]]) - Returns a JSON array string with NULL
elements dropped.",
+ arguments = """
+ Arguments:
+ * expr - the elements to place in the array.
+ """,
+ examples = """
+ Examples:
+ > SELECT _FUNC_(1, 'x', true);
+ [1,"x",true]
+ > SELECT _FUNC_(1, NULL, 3);
+ [1,3]
+ > SELECT _FUNC_();
+ []
+ """,
+ since = "4.4.0",
+ group = "json_funcs")
+object JsonArrayExpressionBuilder extends ExpressionBuilder {
+ override def build(funcName: String, expressions: Seq[Expression]):
Expression = {
+ // A routed call carries no lexical FORMAT JSON, so every element is a
plain value (quoted).
+ // Splicing a nested constructor is only reachable via `JSON_ARRAY(...)`
syntax (which freezes
+ // the decision lexically).
+ // TODO(SPARK-59243): splice nested constructors reached through
routed/qualified calls.
+ val flags = expressions.map(_ => false)
+ JsonArray(expressions, flags, flags, JsonConstructorNullBehavior.Absent,
StringType)
+ }
+}
+
+/**
+ * Shared builder for the plain `JSON_VALUE` / `JSON_QUERY` / `JSON_EXISTS`
forms. The path must
+ * be a foldable string expression; the parser-only SQL/JSON syntax supplies a
string literal,
+ * while ordinary function-call syntax can reach this builder with any
constant string expression.
+ */
+abstract class JsonPathExpressionBuilder extends ExpressionBuilder {
+ protected def buildWithPath(jsonExpr: Expression, path: String): Expression
+
+ override final def build(funcName: String, expressions: Seq[Expression]):
Expression = {
+ if (expressions.length != 2) {
+ throw QueryCompilationErrors.wrongNumArgsError(funcName, Seq(2),
expressions.length)
+ }
+ val pathExpr = expressions(1)
+ pathExpr.dataType match {
+ case _: StringType if pathExpr.foldable =>
+ // TODO(SPARK-59244): wrap eval() so a foldable-but-throwing path
re-throws as a clean
+ // invalid-argument analysis error naming `path`.
+ val pathValue = pathExpr.eval()
+ if (pathValue == null) {
+ throw QueryCompilationErrors.unexpectedNullError("path", pathExpr)
+ }
+ buildWithPath(expressions.head, pathValue.toString)
+ case _: StringType =>
+ throw QueryCompilationErrors.nonFoldableArgumentError(
+ funcName, "path", pathExpr.dataType)
+ case _ =>
+ throw QueryCompilationErrors.unexpectedInputDataTypeError(
+ funcName, 2, StringType, pathExpr)
+ }
+ }
+}
+
+@ExpressionDescription(
+ usage = "_FUNC_(jsonStr, path) - Extracts a SQL scalar as a string.",
+ arguments = """
+ Arguments:
+ * jsonStr - a JSON string.
+ * path - a SQL/JSON path expression given as a foldable string
expression.
Review Comment:
**Nit (P3):** This description omits an enforced restriction: `JSON_VALUE`
and `JSON_QUERY` reject otherwise-valid wildcard paths such as `$.a[*]`, while
the neighboring `JSON_EXISTS` builder accepts them. Please state the
wildcard-free requirement in the two affected path descriptions so the newly
registered user-facing docs match analysis behavior.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionResolution.scala:
##########
@@ -404,6 +388,94 @@ class FunctionResolution(
}
}
+ /**
+ * Returns whether an unqualified function name reaches `system.builtin`
before any temp or
+ * persistent function in the effective SQL PATH. When a temp or persistent
function shadows the
+ * builtin, special-syntax handling that only applies to Spark's builtins
must not fire, since the
+ * name no longer refers to the builtin -- e.g. rejecting a bare `*` in a
routed JSON constructor
+ * or the `count(tbl.*)` guard. Parser-built `count(*)` is normalized to
`count(1)` in
+ * `AstBuilder` so it skips this probe, but a DataFrame `count("*")` keeps
its star and does reach
+ * the probe during analyzer normalization.
+ */
+ def unqualifiedFunctionResolvesToBuiltinBeforeAnyShadow(functionName:
String): Boolean = {
+ // Walk the PATH in order and stop at the first entry that owns the name.
The default order puts
+ // system.builtin first, so the common case returns on the first entry
with no catalog lookup;
+ // only a custom PATH that lists a persistent catalog ahead of
system.builtin reaches the probe
+ // below (one lookup per such preceding entry, recomputed on each call --
not cached).
+ sqlResolutionPathEntriesForAnalysis.foreach { pathEntry =>
+ val candidate = pathEntry :+ functionName
+ FunctionResolution.sessionNamespaceKind(candidate) match {
+ case
Some(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Builtin) =>
+ return true
+ case Some(org.apache.spark.sql.catalyst.catalog.SessionCatalog.Temp) =>
+ // Honor stored-view temp visibility so this probe picks the same
owner the resolver
+ // would: a temp not captured by the view is hidden here too, just
as the persistent
+ // branch below expands through the view's frozen catalog.
+ if
(v1SessionCatalog.isTemporaryFunctionVisible(FunctionIdentifier(functionName)))
{
+ return false
+ }
+ case None =>
+ if (persistentFunctionExists(candidate)) {
+ return false
+ }
+ }
+ }
+ false
+ }
+
+ /**
+ * Returns true when a function reference resolves to the system built-in
with the requested name.
+ * This mirrors [[resolveFunction]] for special parser/analyzer rewrites
that must run only for
+ * Spark's built-ins. In particular, two-part `builtin.name` is not always a
system built-in:
+ * with `spark.sql.legacy.persistentCatalogFirst=true`, an existing
persistent
+ * `current_catalog.builtin.name` takes precedence.
+ */
+ def functionNameResolvesToBuiltin(nameParts: Seq[String], expectedName:
String): Boolean = {
+ if (!FunctionRegistry.functionSet.contains(
+ FunctionRegistry.builtinFunctionIdentifier(expectedName)) ||
+ !FunctionResolution.isUnqualifiedOrBuiltinFunctionName(nameParts,
expectedName)) {
+ return false
+ }
+ nameParts.length match {
+ case 1 =>
+ unqualifiedFunctionResolvesToBuiltinBeforeAnyShadow(nameParts.head)
+ case 2 =>
+ conf.prioritizeSystemCatalog || !persistentFunctionExists(nameParts)
+ case 3 =>
+ true
+ case _ =>
+ false
+ }
+ }
+
+ // All routed SQL/JSON constructors forbid a bare `*` argument. Derived from
the single registry
Review Comment:
**Nit (P3):** Only `JSON_ARRAY` is a constructor; `JSON_VALUE` and
`JSON_QUERY` are path functions and `JSON_EXISTS` is a predicate. Please call
this the routed SQL/JSON function set (or constructor/path-function set) here
and in the corresponding JsonValueSuite comment so the name matches all four
members.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala:
##########
@@ -466,18 +492,272 @@ class JsonArraySuite extends QueryTest with
SharedSparkSession {
}
}
+ test("plain call goes through routine resolution and can be shadowed via SET
PATH") {
+ // `withUserDefinedFunction` is unusable here: its cleanup asserts the
name no longer resolves,
+ // but `json_array` is now a registered built-in, so drop the temporary
routine explicitly.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_array(a INT, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_value(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_query(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_exists(a STRING, b STRING) RETURNS
BOOLEAN " +
+ "RETURN false")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine
(ahead of
+ // system.builtin on the path) shadows the built-in constructor.
+ checkAnswer(sql("SELECT json_array(1, 'x')"), Row("shadowed"))
+ checkAnswer(sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a,
b)"), Row("shadowed"))
+ // The clause-bearing form is not a function call, so it stays the
built-in constructor.
+ checkAnswer(sql("SELECT json_array('x' NULL ON NULL)"),
Row("""["x"]"""))
+ // Nested JSON-producing children stay on the direct-construction
path, so they are not
+ // shadowed. This preserves JSON_ARRAY's parse-time splice decisions.
+ checkAnswer(sql("SELECT json_array(json_array(1))"), Row("[[1]]"))
+ checkAnswer(
+ sql("""SELECT json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""[{"x":1}]"""))
+ // Plain scalar and predicate children are still ordinary function
calls. Use an explicit
+ // outer NULL clause to keep the parent on the direct path while the
children are shadowed.
+ checkAnswer(
+ sql("""SELECT json_array(json_value('{"a":"x"}', '$.a') NULL ON
NULL)"""),
+ Row("""["shadowed"]"""))
+ checkAnswer(
+ sql("""SELECT json_array(json_exists('{"a":1}', '$.a') NULL ON
NULL)"""),
+ Row("[false]"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_array")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_value")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_query")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_exists")
+ }
+ }
+ }
+
+ test("qualified plain JSON_ARRAY resolves to the built-in constructor") {
+ checkAnswer(sql("SELECT builtin.json_array(1, 'x')"), Row("""[1,"x"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(1, 'x')"),
Row("""[1,"x"]"""))
+ }
+
+ test("a nested JSON constructor through a routed JSON_ARRAY call is quoted,
not spliced") {
+ // A routed (plain or qualified) call carries no lexical FORMAT JSON, so a
nested JSON
+ // constructor argument is treated as a plain value and quoted, unlike the
JSON_ARRAY(...)
+ // grammar which splices it (see the unqualified
`json_array(json_array(1))` -> `[[1]]` cases
+ // above). A nested constructor reaches the routed builder only via a
qualified outer call:
Review Comment:
**Nit (P3):** Outer qualification is not required here.
`json_array(builtin.json_array(1))` also reaches the routed outer builder
because the qualified child is still unresolved when `hasImplicitJson` is
computed, and it produces the quoted form. Please describe the actual lexical
condition rather than limiting this path to a qualified outer call.
##########
sql/core/src/test/scala/org/apache/spark/sql/JsonArraySuite.scala:
##########
@@ -466,18 +492,272 @@ class JsonArraySuite extends QueryTest with
SharedSparkSession {
}
}
+ test("plain call goes through routine resolution and can be shadowed via SET
PATH") {
+ // `withUserDefinedFunction` is unusable here: its cleanup asserts the
name no longer resolves,
+ // but `json_array` is now a registered built-in, so drop the temporary
routine explicitly.
+ withSQLConf(
+ SQLConf.PATH_ENABLED.key -> "true",
+ SQLConf.SESSION_FUNCTION_RESOLUTION_ORDER.key -> "second") {
+ try {
+ sql("CREATE TEMPORARY FUNCTION json_array(a INT, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_value(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_query(a STRING, b STRING) RETURNS
STRING " +
+ "RETURN 'shadowed'")
+ sql("CREATE TEMPORARY FUNCTION json_exists(a STRING, b STRING) RETURNS
BOOLEAN " +
+ "RETURN false")
+ sql("SET PATH = system.session, system.builtin")
+ // A plain call is an ordinary function call, so the temporary routine
(ahead of
+ // system.builtin on the path) shadows the built-in constructor.
+ checkAnswer(sql("SELECT json_array(1, 'x')"), Row("shadowed"))
+ checkAnswer(sql("SELECT json_array(*) FROM VALUES (1, 'x') AS t(a,
b)"), Row("shadowed"))
+ // The clause-bearing form is not a function call, so it stays the
built-in constructor.
+ checkAnswer(sql("SELECT json_array('x' NULL ON NULL)"),
Row("""["x"]"""))
+ // Nested JSON-producing children stay on the direct-construction
path, so they are not
+ // shadowed. This preserves JSON_ARRAY's parse-time splice decisions.
+ checkAnswer(sql("SELECT json_array(json_array(1))"), Row("[[1]]"))
+ checkAnswer(
+ sql("""SELECT json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""[{"x":1}]"""))
+ // Plain scalar and predicate children are still ordinary function
calls. Use an explicit
+ // outer NULL clause to keep the parent on the direct path while the
children are shadowed.
+ checkAnswer(
+ sql("""SELECT json_array(json_value('{"a":"x"}', '$.a') NULL ON
NULL)"""),
+ Row("""["shadowed"]"""))
+ checkAnswer(
+ sql("""SELECT json_array(json_exists('{"a":1}', '$.a') NULL ON
NULL)"""),
+ Row("[false]"))
+ } finally {
+ sql("SET PATH = DEFAULT_PATH")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_array")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_value")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_query")
+ sql("DROP TEMPORARY FUNCTION IF EXISTS json_exists")
+ }
+ }
+ }
+
+ test("qualified plain JSON_ARRAY resolves to the built-in constructor") {
+ checkAnswer(sql("SELECT builtin.json_array(1, 'x')"), Row("""[1,"x"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(1, 'x')"),
Row("""[1,"x"]"""))
+ }
+
+ test("a nested JSON constructor through a routed JSON_ARRAY call is quoted,
not spliced") {
+ // A routed (plain or qualified) call carries no lexical FORMAT JSON, so a
nested JSON
+ // constructor argument is treated as a plain value and quoted, unlike the
JSON_ARRAY(...)
+ // grammar which splices it (see the unqualified
`json_array(json_array(1))` -> `[[1]]` cases
+ // above). A nested constructor reaches the routed builder only via a
qualified outer call:
+ // an unqualified nested constructor stays on the direct grammar path.
Splicing through a
+ // routed call is left as a follow-up.
+ checkAnswer(sql("SELECT builtin.json_array(json_array(1))"),
Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT system.builtin.json_array(json_array(1))"),
Row("""["[1]"]"""))
+ checkAnswer(sql("SELECT builtin.json_array(json_array(1), 2)"),
Row("""["[1]",2]"""))
+ checkAnswer(
+ sql("""SELECT builtin.json_array(json_query('{"a":{"x":1}}', '$.a'))"""),
+ Row("""["{\"x\":1}"]"""))
+ }
+
+ test("invalid: a bare star argument in plain JSON_ARRAY is not expanded") {
+ Seq("json_array", "builtin.json_array",
"system.builtin.json_array").foreach { func =>
+ val e = intercept[AnalysisException] {
+ sql(s"SELECT $func(*) FROM VALUES (1, 'x') AS t(a, b)").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX", s"for
$func(*)")
+ }
+ }
+
+ test("invalid: a bare star argument in clause-bearing JSON_ARRAY is not
expanded") {
+ val e = intercept[AnalysisException] {
+ sql("SELECT json_array(* NULL ON NULL) FROM VALUES (1, 'x') AS t(a,
b)").collect()
+ }
+ assert(e.getCondition == "INVALID_USAGE_OF_STAR_OR_REGEX")
+ }
+
+ test("JSON_ARRAY expands a star nested in a sibling constructor (array(*))")
{
+ // Only a bare `*` element is rejected. A star nested in `array(...)`
belongs to that call and
+ // is expanded there, exactly as `array(array(*))` would, then JSON_ARRAY
wraps the result.
+ checkAnswer(
+ sql("SELECT json_array(array(*)) FROM VALUES (1, 2) AS t(a, b)"),
+ Row("[[1,2]]"))
+ // Clause-bearing form (a direct-construction JsonArray node) behaves the
same.
+ checkAnswer(
+ sql("SELECT json_array(array(*) NULL ON NULL) FROM VALUES (1, 2) AS t(a,
b)"),
+ Row("[[1,2]]"))
+ // Alongside count(*): the array's star expands, count(*) is rewritten,
neither is rejected.
+ checkAnswer(
+ sql("SELECT json_array(count(*), array(max(a))) FROM VALUES (1), (2) AS
t(a)"),
+ Row("[2,[2]]"))
+ }
+
+ test("single-pass: JSON_ARRAY expands a star nested in array(*)") {
+ withSQLConf(SQLConf.ANALYZER_SINGLE_PASS_RESOLVER_ENABLED.key -> "true") {
+ Seq(
+ "SELECT json_array(array(*)) FROM VALUES (1, 2) AS t(a, b)",
+ "SELECT json_array(array(*) NULL ON NULL) FROM VALUES (1, 2) AS t(a,
b)"
+ ).foreach { query =>
+ // Analyze only: the single-pass analyzer cannot execute every
operator, so assert the
Review Comment:
**Nit (P3):** The single-pass analyzer does not execute operators; this test
stops at `queryExecution.analyzed` because it cannot yet analyze or resolve
every operator required by the action path. Please use analysis/resolution
terminology in both copies of this comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
