cloud-fan commented on code in PR #58034: URL: https://github.com/apache/spark/pull/58034#discussion_r4053382886
########## docs/sql-ref-syntax-qry-select-json-object.md: ########## @@ -0,0 +1,144 @@ +--- +layout: global +title: JSON_OBJECT +displayTitle: JSON_OBJECT +license: | + Licensed to the Apache Software Foundation (ASF) under one or more + contributor license agreements. See the NOTICE file distributed with + this work for additional information regarding copyright ownership. + The ASF licenses this file to You under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with + the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. +--- + +### Description + +The `JSON_OBJECT` constructor function builds a JSON object from key/value pairs and returns it as +JSON text. This is the SQL-standard way (SQL:2016) to assemble a JSON object inline, with common +compatibility syntax from other systems. `JSON_OBJECT` is an expression that can appear anywhere a +value is allowed. + +Each value is serialized with the same JSON writer as the built-in `to_json` function, so numbers, +decimals, booleans, dates, timestamps, and nested structs/arrays/maps render the same way. +Null-field handling inside a struct value therefore follows +`spark.sql.jsonGenerator.ignoreNullFields`, exactly as `to_json` does; the `ON NULL` clause below +controls only the top-level object members. + +This is an initial subset of the SQL:2016 `JSON_OBJECT` constructor. It supports the key/value +members, the `{ NULL | ABSENT } ON NULL` clause, and a string-type `RETURNING`. The following +SQL/JSON clauses are not yet supported: + +* The value-level `FORMAT JSON` marker (which tags a string value as pre-formatted JSON to be + spliced in raw). A nested `JSON_OBJECT` is still spliced in as raw JSON; see the **value** Review Comment: **Nit (P3):** This reads as an unconditional promise that nested constructors are spliced as JSON, but qualified and otherwise routed calls currently quote the nested result (for example, `builtin.json_object('a', json_object('b', 1))`). Please qualify the statement and examples so users can distinguish direct nested syntax from the routed behavior that is deferred to SPARK-59243. ########## sql/core/src/test/scala/org/apache/spark/sql/JsonObjectSuite.scala: ########## @@ -0,0 +1,778 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql + +import org.apache.spark.SparkRuntimeException +import org.apache.spark.sql.catalyst.analysis.Star +import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch +import org.apache.spark.sql.catalyst.expressions.{Cast, Collate, JsonConstructorNullBehavior, JsonObjectExpr, Literal} +import org.apache.spark.sql.catalyst.parser.ParseException +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types.{CharType, GeometryType, IntegerType, MapType, StringType, VarcharType} + +/** + * End-to-end tests for the SQL:2016 `JSON_OBJECT` constructor function. + */ +class JsonObjectSuite extends QueryTest with SharedSparkSession { + import testImplicits._ + + test("basic object from key-value pairs using VALUE keyword") { + checkAnswer( + sql("SELECT json_object('id' VALUE 7, 'name' VALUE 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("construct object using optional KEY keyword") { + checkAnswer( + sql("SELECT json_object(KEY 'id' VALUE 7, KEY 'name' VALUE 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("construct object using colon syntax") { + checkAnswer( + sql("SELECT json_object('id': 7, 'name': 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("construct object using comma-separated key-value syntax") { + checkAnswer( + sql("SELECT json_object('id', 7, 'name', 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("an odd number of arguments in the comma syntax is rejected") { + // The comma form requires paired key/value arguments; a dangling key ('name') has no value. + // JSON_OBJECT is a non-reserved keyword, so when the constructor grammar cannot match, the call + // parses as an ordinary function call and routes to the registered built-in, whose builder + // rejects the odd argument count rather than silently dropping the dangling key. + val e = intercept[AnalysisException] { + sql("SELECT json_object('id', 7, 'name')") + } + assert(e.getCondition == "WRONG_NUM_ARGS.WITHOUT_SUGGESTION") + } + + test("mixing the VALUE/colon form and the comma form is a parse error") { + // The two member-list styles are mutually exclusive grammar alternatives, so a single + // constructor cannot mix `key VALUE value` (or `key : value`) members with `key, value` ones. + Seq( + "SELECT json_object('a', 1, 'b' VALUE 2)", + "SELECT json_object('a' VALUE 1, 'b', 2)", + "SELECT json_object('a' : 1, 'b', 2)").foreach { query => + intercept[ParseException](sql(query)) + } + } + + test("construct object with NULL values (default NULL ON NULL)") { + checkAnswer( + sql("SELECT json_object('id': 7, 'v': NULL)"), + Row("""{"id":7,"v":null}""")) + } + + test("construct object with explicit NULL ON NULL") { + checkAnswer( + sql("SELECT json_object('id', 7, 'v', NULL NULL ON NULL)"), + Row("""{"id":7,"v":null}""")) + } + + test("construct object with NULL values and ABSENT ON NULL") { + checkAnswer( + sql("SELECT json_object('id': 7, 'v': NULL ABSENT ON NULL)"), + Row("""{"id":7}""")) + } + + test("construct empty object") { + checkAnswer( + sql("SELECT json_object()"), + Row("{}")) + } + + test("construct object with mixed scalar types") { + checkAnswer( + sql("""SELECT json_object('int': 42, 'str': 'hello', 'bool': true, + 'float': 3.14)"""), + Row("""{"int":42,"str":"hello","bool":true,"float":3.14}""")) + } + + test("construct object with decimal type via Jackson") { + checkAnswer( + sql("""SELECT json_object('d' VALUE CAST('123.45' AS DECIMAL(5,2)))"""), + Row("""{"d":123.45}""")) + } + + test("construct object with DATE type via Jackson") { + checkAnswer( + sql("""SELECT json_object('d' VALUE DATE'2020-01-02')"""), + Row("""{"d":"2020-01-02"}""")) + } + + test("construct object with TIMESTAMP type via Jackson") { + // Note: Jackson includes timezone offset when session timezone is set + checkAnswer( + sql("""SELECT json_object('ts' VALUE TIMESTAMP'2020-01-02 10:30:00')"""), + Row("""{"ts":"2020-01-02T10:30:00.000-08:00"}""")) + } + + test("struct value renders like to_json") { + // A struct value must render exactly like `to_json` of the equivalent member. + checkAnswer( + sql("SELECT json_object('s' VALUE named_struct('a', 1, 'b', 'x'))"), + Row("""{"s":{"a":1,"b":"x"}}""")) + checkAnswer( + sql("SELECT json_object('s' VALUE named_struct('a', 1, 'b', 'x'))"), + sql("SELECT to_json(named_struct('s', named_struct('a', 1, 'b', 'x')))")) + } + + test("array value renders like to_json") { + checkAnswer( + sql("SELECT json_object('a' VALUE array(1, 2, 3))"), + Row("""{"a":[1,2,3]}""")) + checkAnswer( + sql("SELECT json_object('a' VALUE array(1, 2, 3))"), + sql("SELECT to_json(named_struct('a', array(1, 2, 3)))")) + } + + test("map value renders like to_json") { + checkAnswer( + sql("SELECT json_object('m' VALUE map('x', 1, 'y', 2))"), + Row("""{"m":{"x":1,"y":2}}""")) + checkAnswer( + sql("SELECT json_object('m' VALUE map('x', 1, 'y', 2))"), + sql("SELECT to_json(named_struct('m', map('x', 1, 'y', 2)))")) + } + + test("nested complex value combining struct, array and map renders like to_json") { + val value = "named_struct('arr', array(1, 2), 'm', map('k', named_struct('n', 3)))" + checkAnswer( + sql(s"SELECT json_object('c' VALUE $value)"), + sql(s"SELECT to_json(named_struct('c', $value))")) + } + + test("struct value honors spark.sql.jsonGenerator.ignoreNullFields like to_json") { + // `ON NULL` controls only top-level members; a null field *inside* a struct value follows + // spark.sql.jsonGenerator.ignoreNullFields, like `to_json`. + val value = "named_struct('a', 1, 'b', CAST(NULL AS INT))" + Seq("true", "false").foreach { ignore => + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> ignore) { + checkAnswer( + sql(s"SELECT json_object('s' VALUE $value)"), + sql(s"SELECT to_json(named_struct('s', $value))")) + } + } + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> "false") { + checkAnswer(sql(s"SELECT json_object('s' VALUE $value)"), Row("""{"s":{"a":1,"b":null}}""")) + } + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> "true") { + checkAnswer(sql(s"SELECT json_object('s' VALUE $value)"), Row("""{"s":{"a":1}}""")) + } + } + + test("top-level ON NULL and struct-internal ignoreNullFields are independent") { + // With NULL ON NULL (default) and ignoreNullFields=true, a top-level NULL member is kept as + // `null` while a null field inside a struct value is dropped. + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> "true") { + checkAnswer( + sql("""SELECT json_object('top' VALUE CAST(NULL AS INT), + 's' VALUE named_struct('a', 1, 'b', CAST(NULL AS INT)))"""), + Row("""{"top":null,"s":{"a":1}}""")) + } + } + + test("string escaping in keys") { + checkAnswer( + sql("""SELECT json_object('key"with"quotes' VALUE 1)"""), + Row("""{"key\"with\"quotes":1}""")) + } + + // For scalar string values JSON_OBJECT must escape exactly like to_json of the equivalent + // struct (both go through the same Jackson generator); assert that equivalence rather than + // hand-encoding the escaping, which is easy to get wrong across Scala/SQL/JSON layers. + test("string escaping in values matches to_json") { + checkAnswer( + sql("""SELECT json_object('msg' VALUE 'hello +world')"""), + sql("""SELECT to_json(named_struct('msg', 'hello +world'))""")) + } + + test("string escaping with backslash matches to_json") { + checkAnswer( + sql("""SELECT json_object('path' VALUE 'c:\windows')"""), + sql("""SELECT to_json(named_struct('path', 'c:\windows'))""")) + } + + test("nested JSON_OBJECT spliced raw") { + checkAnswer( + sql("""SELECT json_object('a' VALUE json_object('b' VALUE 1))"""), + Row("""{"a":{"b":1}}""")) + checkAnswer( + sql("""SELECT json_object('a', json_object('b', 1))"""), + Row("""{"a":{"b":1}}""")) + } + + test("nested JSON_OBJECT with multiple levels") { + checkAnswer( + sql("""SELECT json_object('outer' VALUE + json_object('inner' VALUE 42, 'name' VALUE 'test'))"""), + Row("""{"outer":{"inner":42,"name":"test"}}""")) + } + + test("a nested JSON_ARRAY value is spliced raw") { + checkAnswer( + sql("SELECT json_object('a' VALUE json_array(1, 2))"), + Row("""{"a":[1,2]}""")) + } + + test("JSON_OBJECT nested directly in JSON_ARRAY is spliced as an object element") { + // The inverse nesting direction: a JSON_OBJECT in a JSON_ARRAY element position stays on the + // direct grammar path (JsonArrayValueContext), so it is spliced as a JSON object rather than + // routed through resolution and emitted as a quoted string. + checkAnswer( + sql("SELECT json_array(json_object('a', 1), json_object('b', 2))"), + Row("""[{"a":1},{"b":2}]""")) + } + + test("a nested JSON_QUERY value is spliced under KEEP QUOTES and quoted under OMIT QUOTES") { + // JSON_QUERY emits JSON text under the default KEEP QUOTES, so a lexically nested JSON_QUERY is + // spliced raw: the matched object is {"x":1}, not the quoted string "{\"x\":1}". + checkAnswer( + sql("""SELECT json_object('a' VALUE json_query('{"o":{"x":1}}', '$.o'))"""), + Row("""{"a":{"x":1}}""")) + // OMIT QUOTES returns the matched scalar string's decoded content (Ada, not "Ada") -- an + // ordinary string -- so it takes the quoted path (emitsImplicitJsonText is false), never the + // invalid splice {"a":Ada}. + checkAnswer( + sql("""SELECT json_object('a' VALUE json_query('{"n":"Ada"}', '$.n' OMIT QUOTES))"""), + Row("""{"a":"Ada"}""")) + } + + test("null key error") { + val e = intercept[SparkRuntimeException] { + sql("SELECT json_object(NULL VALUE 'value')").collect() + } + // Assert the structured error contract, not just the message text. + assert(e.getCondition == "JSON_OBJECT_NULL_KEY") + assert(e.getSqlState == "2200E") + } + + test("a null key is validated before a null value is omitted under ABSENT ON NULL") { + // ABSENT ON NULL omits members with a null value, but the key is validated first, so a null key + // still raises JSON_OBJECT_NULL_KEY rather than being silently dropped along with the member. + val e = intercept[SparkRuntimeException] { + sql("SELECT json_object(NULL VALUE NULL ABSENT ON NULL)").collect() + } + assert(e.getCondition == "JSON_OBJECT_NULL_KEY") + assert(e.getSqlState == "2200E") + } + + test("non-foldable key and value expressions") { + val df = Seq(("key1", "val1"), ("key2", "val2")).toDF("k", "v") + checkAnswer( + df.selectExpr("json_object(k VALUE v)"), + Seq(Row("""{"key1":"val1"}"""), Row("""{"key2":"val2"}"""))) + } + + test("non-foldable with NULL value and NULL ON NULL") { + val df = Seq(("k", null), ("key", "val")).toDF("k", "v") + checkAnswer( + df.selectExpr("json_object(k VALUE v)"), + Seq(Row("""{"k":null}"""), Row("""{"key":"val"}"""))) + } + + test("non-foldable with NULL value and ABSENT ON NULL") { + val df = Seq(("k", null), ("key", "val")).toDF("k", "v") + checkAnswer( + df.selectExpr("json_object(k VALUE v ABSENT ON NULL)"), + Seq(Row("{}"), Row("""{"key":"val"}"""))) + } + + test("multiple keys with ABSENT ON NULL") { + checkAnswer( + sql("""SELECT json_object('a' VALUE 1, 'b' VALUE NULL, 'c' VALUE 3 + ABSENT ON NULL)"""), + Row("""{"a":1,"c":3}""")) + } + + test("duplicate keys are emitted in source order") { + checkAnswer( + sql("SELECT json_object('k' VALUE 1, 'k' VALUE 2)"), + Row("""{"k":1,"k":2}""")) + } + + test("non-string key type is rejected at analysis, not at execution") { + val ex = intercept[AnalysisException] { + sql("SELECT json_object(1 VALUE 'x')") + } + assert(ex.getMessage.contains("UNEXPECTED_INPUT_TYPE")) + } + + test("non-string key type reports the actual key argument") { + val ex = intercept[AnalysisException] { + sql("SELECT json_object('ok' VALUE 1, 2 VALUE 'bad')") + } + checkError( + exception = ex, + condition = "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE", + sqlState = Some("42K09"), + parameters = Map( + "sqlExpr" -> "\"JSON_OBJECT(ok VALUE 1, 2 VALUE bad)\"", + "paramIndex" -> "third", + "requiredType" -> "\"STRING\"", + "inputSql" -> "\"2\"", + "inputType" -> "\"INT\""), + queryContext = Array(ExpectedContext("json_object('ok' VALUE 1, 2 VALUE 'bad')", 7, 46))) + } + + test("collated STRING RETURNING is accepted") { + // isValidReturningType must accept any StringType instance, not just the default collation. + checkAnswer( + sql("SELECT json_object('a' VALUE 1 RETURNING STRING COLLATE UTF8_LCASE)"), + Row("""{"a":1}""")) + } + + test("an invalid RETURNING type is reported under DATATYPE_MISMATCH") { + // The error is emitted as a DataTypeMismatch, so its condition must resolve under + // DATATYPE_MISMATCH -- not as a top-level INVALID_JSON_RETURNING_TYPE class. + val e = intercept[AnalysisException] { + sql("SELECT json_object('a' VALUE 1 RETURNING INT)").collect() + } + assert(e.getCondition == "DATATYPE_MISMATCH.INVALID_JSON_RETURNING_TYPE") + } + + test("a directly-constructed JsonObjectExpr with a CHAR/VARCHAR RETURNING is rejected") { + // The parser normalizes CHAR/VARCHAR RETURNING to STRING, but a raw CharType/VarcharType from + // direct Catalyst construction would advertise a length JSON_OBJECT does not enforce. + Seq(VarcharType(2), CharType(2)).foreach { returning => + val expr = JsonObjectExpr( + Seq((Literal("k"), Literal(1))), Seq(false), JsonConstructorNullBehavior.Null, returning) + expr.checkInputDataTypes() match { + case DataTypeMismatch(errorSubClass, _) => + assert(errorSubClass == "INVALID_JSON_RETURNING_TYPE", s"for $returning") + case other => fail(s"expected DataTypeMismatch for $returning, got $other") + } + } + } + + test("value accepts an unparenthesized predicate expression") { + // valueExpr is parsed as a full `expression`, so ordinary predicates work without parentheses. + checkAnswer(sql("SELECT json_object('isnull' VALUE 1 IS NULL)"), Row("""{"isnull":false}""")) + checkAnswer(sql("SELECT json_object('gt' : 2 > 1)"), Row("""{"gt":true}""")) + } + + test("widening the value to expression does not change documented forms") { + // Design-doc examples where a value abuts the ON NULL / RETURNING keywords must still parse and + // evaluate identically after widening valueExpression -> expression. + checkAnswer(sql("SELECT json_object('id': 7, 'v': NULL)"), Row("""{"id":7,"v":null}""")) + checkAnswer( + sql("SELECT json_object('id': 7, 'v': NULL ABSENT ON NULL)"), Row("""{"id":7}""")) + checkAnswer( + sql("SELECT json_object('id', 7, 'v', NULL ABSENT ON NULL)"), Row("""{"id":7}""")) + checkAnswer( + sql("SELECT json_object('id' VALUE 7, 'name' VALUE 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("an unsupported value type is rejected at analysis") { Review Comment: **Non-blocking (P2):** These cases cover a top-level spatial value and an allowed spatial map key, but no rejected value requires recursive descent. Please add one focused nested container case, such as a spatial map value, that must fail with `CANNOT_CONVERT_TO_JSON`, while retaining the allowed-key assertion. ########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala: ########## @@ -2210,3 +2245,353 @@ case class JsonTypeof(child: Expression) override protected def withNewChildInternal(newChild: Expression): JsonTypeof = copy(child = newChild) } + +/** + * The SQL:2016 `JSON_OBJECT` constructor function (feature T811): constructs a JSON object from + * key-value pairs, written `key VALUE value`, `KEY key VALUE value`, `key : value`, or the + * MySQL-style `key, value`. + * + * Keys must be non-null strings; a null key is an error. The `ON NULL` clause controls whether + * null-valued pairs are included (NULL ON NULL, the standard default) or omitted (ABSENT ON NULL). + * + * A value is spliced in raw (unquoted) when it is a lexically nested `ImplicitlyFormattedAsJson` + * producer whose `emitsImplicitJsonText` holds -- a nested JSON constructor, or `JSON_QUERY` under + * KEEP QUOTES -- tracked per member by `rawJson` (see `rawJsonValue`). The standard's explicit + * value-level `FORMAT JSON` clause (e.g. `JSON_OBJECT('a' VALUE '{"b":1}' FORMAT JSON)`) is + * deferred. + * + * Examples: + * JSON_OBJECT('id' VALUE 7, 'name' VALUE 'Ada') -> '{"id":7,"name":"Ada"}' + * JSON_OBJECT('id': 7, 'v': NULL) -> '{"id":7,"v":null}' + * JSON_OBJECT('id': 7, 'v': NULL ABSENT ON NULL) -> '{"id":7}' + * JSON_OBJECT() -> '{}' + * + * A flat, clause-free call routes through function resolution and is rebuilt by + * `JsonObjectExpressionBuilder` (so a same-named routine can shadow the built-in `json_object`); a + * clause-bearing or nested form is built directly from the grammar (see + * `AstBuilder.visitJsonObject`). The user-facing reference lives in + * `docs/sql-ref-syntax-qry-select-json-object.md`. + */ +case class JsonObjectExpr( + members: Seq[(Expression, Expression)], + rawJson: Seq[Boolean], + nullBehavior: JsonConstructorNullBehavior = JsonConstructorNullBehavior.Null, + returning: DataType = StringType, + timeZoneId: Option[String] = None) + extends Expression + with TimeZoneAwareExpression + with CodegenFallback + with QueryErrorsBase + // Default RETURNING is a plain STRING, so `DefaultStringProducingExpression` lets + // `ApplyDefaultCollation` cast the result to a non-default collation; the `dataType` override + // below stays authoritative when RETURNING is given explicitly. + with DefaultStringProducingExpression + with ImplicitlyFormattedAsJson { + + // `rawJson(i)` marks member `i`'s value as already-JSON text to splice verbatim, not quote; it is + // frozen at parse time (see `AstBuilder.visitJsonObject`, [[ImplicitlyFormattedAsJson]]). + require(members.length == rawJson.length, + "JsonObjectExpr requires one rawJson flag per member") + + @transient private lazy val memberArray: Array[(Expression, Expression)] = + members.toArray + + @transient private lazy val rawJsonArray: Array[Boolean] = + rawJson.toArray + + // Always throwable: a null key raises JSON_OBJECT_NULL_KEY at eval even with non-throwable + // children, which keeps the optimizer from pushing it below a filtering join. Left non-foldable + // (the default) so folding a constant call does not eagerly raise that null-key error at + // optimization for rows a filter would later drop. + override lazy val throwable: Boolean = true + + // The value is never null, but report nullable as `throwable` (like JsonArray): `NullPropagation` + // keys off `nullable`, so a non-nullable constructor would let it fold `IS [NOT] NULL` to a + // constant or `count(...)` to `count(1)`, skipping the eval that must raise JSON_OBJECT_NULL_KEY. + override def nullable: Boolean = throwable + + override def dataType: DataType = returning + + // Flattened `[k0, v0, k1, v1, ...]` view, cached since `children` is walked repeatedly. + @transient private lazy val childrenSeq: Seq[Expression] = + memberArray.flatMap { case (k, v) => Seq(k, v) }.toImmutableArraySeq + + override def children: Seq[Expression] = childrenSeq + + override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression = + copy(timeZoneId = Option(timeZoneId)) + + override def checkInputDataTypes(): TypeCheckResult = { + val inputCheck = super.checkInputDataTypes() + if (inputCheck.isFailure) { + inputCheck + } else if (!JsonObjectExpr.isValidReturningType(returning)) { + DataTypeMismatch( + errorSubClass = "INVALID_JSON_RETURNING_TYPE", + messageParameters = Map( + "functionName" -> toSQLId(prettyName), + "returningType" -> toSQLType(returning))) + } else { + // Keys must be character strings (they become JSON object member names). Reject any other + // concrete type up front so a non-string key cannot reach the StringType Jackson writer and + // fail with an internal ClassCastException. NullType is allowed through: a null key value is + // reported at runtime as JSON_OBJECT_NULL_KEY rather than a type error. + val keyCheck = memberArray.iterator.map(_._1).zipWithIndex.collectFirst { + case (k, keyIndex) if !(k.dataType.isInstanceOf[StringType] || k.dataType == NullType) => + DataTypeMismatch( + errorSubClass = "UNEXPECTED_INPUT_TYPE", + messageParameters = Map( + "paramIndex" -> ordinalNumber(keyIndex * 2), + "requiredType" -> toSQLType(StringType), + "inputSql" -> toSQLExpr(k), + "inputType" -> toSQLType(k.dataType))) + }.getOrElse(TypeCheckResult.TypeCheckSuccess) + if (keyCheck.isFailure) { + keyCheck + } else { + // Every value must be serializable to JSON. This mirrors `to_json`'s analysis-time + // `JacksonUtils.verifyType` check, plus a guard for the spatial atomics that check accepts + // but `JacksonGenerator` cannot write (rejecting a runtime failure at analysis instead). + memberArray.iterator.map(_._2).zipWithIndex + .foldLeft(TypeCheckResult.TypeCheckSuccess: TypeCheckResult) { + case (acc, _) if acc.isFailure => acc + case (_, (v, valueIndex)) if rawJsonArray(valueIndex) && + !(v.dataType.isInstanceOf[StringType] || v.dataType == NullType) => + // `eval` casts a raw-spliced value to UTF8String. The parser only marks a nested + // constructor (STRING-typed) raw, so reject a non-string raw value from direct + // Catalyst construction here instead of failing with a ClassCastException at eval. + DataTypeMismatch( + errorSubClass = "UNEXPECTED_INPUT_TYPE", + messageParameters = Map( + "paramIndex" -> ordinalNumber(valueIndex * 2 + 1), + "requiredType" -> toSQLType(StringType), + "inputSql" -> toSQLExpr(v), + "inputType" -> toSQLType(v.dataType))) + case (_, (v, _)) => + val elemCheck = JacksonUtils.verifyType(prettyName, v.dataType) + if (elemCheck.isFailure) { + elemCheck + } else if (JsonObjectExpr.containsUnsupportedJsonType(v.dataType)) { + DataTypeMismatch( + errorSubClass = "CANNOT_CONVERT_TO_JSON", + messageParameters = Map( + "name" -> toSQLId(prettyName), + "type" -> toSQLType(v.dataType))) + } else { + TypeCheckResult.TypeCheckSuccess + } + } + } + } + } + + override def stateful: Boolean = true + + @transient private lazy val resolvedZoneId: String = + timeZoneId.getOrElse(SQLConf.get.sessionLocalTimeZone) + + // JacksonGenerator (shared with `to_json`, so rendering matches) can only serialize a container, + // not a bare scalar, so each key/value is wrapped in a one-element array whose brackets are + // stripped in renderKey/appendRenderedValue below. + @transient private lazy val keyEvaluator: StructsToJsonEvaluator = + StructsToJsonEvaluator(Map.empty, ArrayType(StringType), Some(resolvedZoneId)) + + @transient private lazy val valueEvaluators: Array[StructsToJsonEvaluator] = + memberArray.map { case (_, v) => + StructsToJsonEvaluator(Map.empty, ArrayType(v.dataType), + Some(resolvedZoneId)) + } + + @transient private lazy val singleElem: Array[Any] = new Array[Any](1) + + @transient private lazy val singleElemData: GenericArrayData = + new GenericArrayData(singleElem) + + @transient private lazy val castInput: GenericInternalRow = + new GenericInternalRow(1) + + @transient private lazy val returningCast: Expression = + Cast(BoundReference(0, StringType, nullable = true), returning, timeZoneId, + EvalMode.ANSI) + + // `evaluateString` returns the Java String directly, skipping `evaluate`'s UTF8String round trip. + // The key is cached or appended whole, so it still materializes the interior substring. + private def renderKey(key: Any): String = { + singleElem(0) = key + val arrJson = keyEvaluator.evaluateString(singleElemData) + arrJson.substring(1, arrJson.length - 1) + } + + // Append the interior of the "[<frag>]" wrapper straight into the builder (no per-value + // substring), mirroring `JsonArray.appendRenderedElement`. + private def appendRenderedValue(sb: java.lang.StringBuilder, idx: Int, value: Any): Unit = { + singleElem(0) = value + val arrJson = valueEvaluators(idx).evaluateString(singleElemData) + sb.append(arrJson, 1, arrJson.length - 1) + } + + // A foldable, non-null key (the common literal-key case, e.g. `JSON_OBJECT('id' VALUE col)`) + // renders to the same JSON member name on every row, so render it once here instead of re-running + // the Jackson writer per row. `renderedKeys(i)` holds that cached name, or `null` when the key + // must be evaluated and rendered per row -- a non-foldable key, or a foldable key that evaluates + // to `null` (still reported as JSON_OBJECT_NULL_KEY at eval). + @transient private lazy val renderedKeys: Array[String] = memberArray.map { case (k, _) => + if (k.foldable) { + val key = k.eval(EmptyRow) + if (key == null) null else renderKey(key) + } else { + null + } + } + + override def eval(input: InternalRow): Any = { + val sb = new java.lang.StringBuilder("{") + var first = true + var i = 0 + val localMembers = memberArray + val localRawJson = rawJsonArray + while (i < localMembers.length) { + val (keyExpr, valueExpr) = localMembers(i) + // Resolve the member's key. A foldable non-null key is already rendered and cached. Otherwise + // evaluate the key before the value so a null key deterministically raises + // JSON_OBJECT_NULL_KEY, independent of whether the value expression happens to throw. + // Delay rendering dynamic keys until we know the member will be emitted, avoiding work for + // null values under ABSENT ON NULL. + val cachedKey = renderedKeys(i) + val key = if (cachedKey == null) { + val key = keyExpr.eval(input) + if (key == null) { + throw QueryExecutionErrors.jsonObjectNullKeyError() + } + key + } else { + null + } + val value = valueExpr.eval(input) + if (value != null || nullBehavior == JsonConstructorNullBehavior.Null) { + if (!first) sb.append(",") + first = false + val keyName = if (cachedKey != null) cachedKey else renderKey(key) + sb.append(keyName).append(":") + if (value == null) { + sb.append("null") + } else if (localRawJson(i)) { + sb.append(value.asInstanceOf[UTF8String].toString) + } else { + appendRenderedValue(sb, i, value) + } + } + i += 1 + } + sb.append("}") + val jsonStr = UTF8String.fromString(sb.toString) + if (returning == StringType) { + jsonStr + } else { + castInput.update(0, jsonStr) + returningCast.eval(castInput) + } + } + + override def prettyName: String = "json_object" + + override def sql: String = { + val membersSQL = members.zip(rawJson).map { case ((k, v), raw) => + // JSON_OBJECT has no value-level FORMAT JSON marker, so rawness can only be expressed in SQL + // by rendering the value as a bare JSON constructor (which reparse re-derives as implicit + // FORMAT JSON). Cover both directions of optimizer rewrites around the frozen `raw` flag: + val valueSQL = (raw, v) match { + // Raw value: a nested constructor, possibly behind collation-only wrappers (an explicit + // COLLATE or a default-collation Cast). Neither affects the raw-spliced bytes, so render + // the bare constructor and reparse re-derives raw splicing. + // TODO(SPARK-59243): a foldable nested value (e.g. JSON_ARRAY(1)) can be constant-folded to + // a string literal; with no value-level FORMAT JSON to restore rawness, this falls back to + // `v.sql` and the emitted SQL reparses quoted. Eval is unaffected (rawJson still splices). + case (true, _) => JsonObjectExpr.rawJsonConstructor(v).map(_.sql).getOrElse(v.sql) Review Comment: **Non-blocking (P2):** After ConstantFolding turns a nested `JSON_ARRAY(1)` into the literal `'[1]'`, `rawJson` remains true but this fallback emits only the literal SQL. Reparsing then records `rawJson=false`, changing `{"a":[1]}` into `{"a":"[1]"}`. Please give canonical member SQL a parseable representation of the frozen rawness decision and add an optimize-render-reparse regression case, without broadening routed or qualified calls to raw splicing. **Recommended change:** Give canonical JSON_OBJECT SQL an explicit, parseable way to preserve each frozen raw member after child rewriting, and add optimizer-sensitive round-trip coverage. **Why this works:** Encode rawJson independently of the current child shape in JsonObjectExpr.sql and teach the parsing path to restore the same flag. If the chosen syntax accepts arbitrary user JSON text, retain the required validation distinction so this repair does not turn trusted implicit producers into an unvalidated public FORMAT JSON path. **Scope:** Align JSON_OBJECT member syntax, AST construction, expression rendering, and round-trip tests around durable raw-member semantics. **Compatibility:** Runtime evaluation of existing JsonObjectExpr trees must keep the same raw-versus-quoted results and routed/qualified calls must continue to quote nested JSON-producing arguments pending SPARK-59243. **Risks:** A new raw-value syntax can expose malformed user text unless trusted implicit producers and explicitly supplied JSON text retain distinct validation semantics. Rendering wrappers around collated or folded children can change bytes, null handling, or error timing unless round-trip cases cover them. **Constraints:** Preserve the accepted direct routing of lexically nested JSON-producing values. Do not broaden qualified or otherwise routed calls to implicit raw splicing, which remains deferred to SPARK-59243. **Success:** After a nested JSON constructor is folded to a literal, JsonObjectExpr.sql reparses to an expression that still splices the value raw. Quoted members remain quoted when their rewritten child later resembles an implicit JSON producer. The rendered SQL stays valid for supported collations and null values without changing runtime validation behavior. ########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala: ########## @@ -2210,3 +2245,353 @@ case class JsonTypeof(child: Expression) override protected def withNewChildInternal(newChild: Expression): JsonTypeof = copy(child = newChild) } + +/** + * The SQL:2016 `JSON_OBJECT` constructor function (feature T811): constructs a JSON object from + * key-value pairs, written `key VALUE value`, `KEY key VALUE value`, `key : value`, or the + * MySQL-style `key, value`. + * + * Keys must be non-null strings; a null key is an error. The `ON NULL` clause controls whether + * null-valued pairs are included (NULL ON NULL, the standard default) or omitted (ABSENT ON NULL). + * + * A value is spliced in raw (unquoted) when it is a lexically nested `ImplicitlyFormattedAsJson` + * producer whose `emitsImplicitJsonText` holds -- a nested JSON constructor, or `JSON_QUERY` under + * KEEP QUOTES -- tracked per member by `rawJson` (see `rawJsonValue`). The standard's explicit + * value-level `FORMAT JSON` clause (e.g. `JSON_OBJECT('a' VALUE '{"b":1}' FORMAT JSON)`) is + * deferred. + * + * Examples: + * JSON_OBJECT('id' VALUE 7, 'name' VALUE 'Ada') -> '{"id":7,"name":"Ada"}' + * JSON_OBJECT('id': 7, 'v': NULL) -> '{"id":7,"v":null}' + * JSON_OBJECT('id': 7, 'v': NULL ABSENT ON NULL) -> '{"id":7}' + * JSON_OBJECT() -> '{}' + * + * A flat, clause-free call routes through function resolution and is rebuilt by + * `JsonObjectExpressionBuilder` (so a same-named routine can shadow the built-in `json_object`); a + * clause-bearing or nested form is built directly from the grammar (see Review Comment: **Nit (P3):** `nested form` is broader than the implemented routing rule: value-position JSON producers stay on the direct grammar path, but a JSON_OBJECT used in key position can still resolve through the routine path and be shadowed. Please state that positional distinction explicitly. ########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala: ########## @@ -2210,3 +2245,353 @@ case class JsonTypeof(child: Expression) override protected def withNewChildInternal(newChild: Expression): JsonTypeof = copy(child = newChild) } + +/** + * The SQL:2016 `JSON_OBJECT` constructor function (feature T811): constructs a JSON object from + * key-value pairs, written `key VALUE value`, `KEY key VALUE value`, `key : value`, or the + * MySQL-style `key, value`. + * + * Keys must be non-null strings; a null key is an error. The `ON NULL` clause controls whether + * null-valued pairs are included (NULL ON NULL, the standard default) or omitted (ABSENT ON NULL). + * + * A value is spliced in raw (unquoted) when it is a lexically nested `ImplicitlyFormattedAsJson` + * producer whose `emitsImplicitJsonText` holds -- a nested JSON constructor, or `JSON_QUERY` under + * KEEP QUOTES -- tracked per member by `rawJson` (see `rawJsonValue`). The standard's explicit + * value-level `FORMAT JSON` clause (e.g. `JSON_OBJECT('a' VALUE '{"b":1}' FORMAT JSON)`) is + * deferred. + * + * Examples: + * JSON_OBJECT('id' VALUE 7, 'name' VALUE 'Ada') -> '{"id":7,"name":"Ada"}' + * JSON_OBJECT('id': 7, 'v': NULL) -> '{"id":7,"v":null}' + * JSON_OBJECT('id': 7, 'v': NULL ABSENT ON NULL) -> '{"id":7}' + * JSON_OBJECT() -> '{}' + * + * A flat, clause-free call routes through function resolution and is rebuilt by + * `JsonObjectExpressionBuilder` (so a same-named routine can shadow the built-in `json_object`); a + * clause-bearing or nested form is built directly from the grammar (see + * `AstBuilder.visitJsonObject`). The user-facing reference lives in + * `docs/sql-ref-syntax-qry-select-json-object.md`. + */ +case class JsonObjectExpr( + members: Seq[(Expression, Expression)], + rawJson: Seq[Boolean], + nullBehavior: JsonConstructorNullBehavior = JsonConstructorNullBehavior.Null, + returning: DataType = StringType, + timeZoneId: Option[String] = None) + extends Expression + with TimeZoneAwareExpression + with CodegenFallback + with QueryErrorsBase + // Default RETURNING is a plain STRING, so `DefaultStringProducingExpression` lets + // `ApplyDefaultCollation` cast the result to a non-default collation; the `dataType` override + // below stays authoritative when RETURNING is given explicitly. + with DefaultStringProducingExpression + with ImplicitlyFormattedAsJson { + + // `rawJson(i)` marks member `i`'s value as already-JSON text to splice verbatim, not quote; it is + // frozen at parse time (see `AstBuilder.visitJsonObject`, [[ImplicitlyFormattedAsJson]]). + require(members.length == rawJson.length, + "JsonObjectExpr requires one rawJson flag per member") + + @transient private lazy val memberArray: Array[(Expression, Expression)] = + members.toArray + + @transient private lazy val rawJsonArray: Array[Boolean] = + rawJson.toArray + + // Always throwable: a null key raises JSON_OBJECT_NULL_KEY at eval even with non-throwable + // children, which keeps the optimizer from pushing it below a filtering join. Left non-foldable + // (the default) so folding a constant call does not eagerly raise that null-key error at + // optimization for rows a filter would later drop. + override lazy val throwable: Boolean = true + + // The value is never null, but report nullable as `throwable` (like JsonArray): `NullPropagation` + // keys off `nullable`, so a non-nullable constructor would let it fold `IS [NOT] NULL` to a + // constant or `count(...)` to `count(1)`, skipping the eval that must raise JSON_OBJECT_NULL_KEY. + override def nullable: Boolean = throwable + + override def dataType: DataType = returning + + // Flattened `[k0, v0, k1, v1, ...]` view, cached since `children` is walked repeatedly. + @transient private lazy val childrenSeq: Seq[Expression] = + memberArray.flatMap { case (k, v) => Seq(k, v) }.toImmutableArraySeq + + override def children: Seq[Expression] = childrenSeq + + override def withTimeZone(timeZoneId: String): TimeZoneAwareExpression = + copy(timeZoneId = Option(timeZoneId)) + + override def checkInputDataTypes(): TypeCheckResult = { + val inputCheck = super.checkInputDataTypes() + if (inputCheck.isFailure) { + inputCheck + } else if (!JsonObjectExpr.isValidReturningType(returning)) { + DataTypeMismatch( + errorSubClass = "INVALID_JSON_RETURNING_TYPE", + messageParameters = Map( + "functionName" -> toSQLId(prettyName), + "returningType" -> toSQLType(returning))) + } else { + // Keys must be character strings (they become JSON object member names). Reject any other + // concrete type up front so a non-string key cannot reach the StringType Jackson writer and + // fail with an internal ClassCastException. NullType is allowed through: a null key value is + // reported at runtime as JSON_OBJECT_NULL_KEY rather than a type error. + val keyCheck = memberArray.iterator.map(_._1).zipWithIndex.collectFirst { + case (k, keyIndex) if !(k.dataType.isInstanceOf[StringType] || k.dataType == NullType) => + DataTypeMismatch( + errorSubClass = "UNEXPECTED_INPUT_TYPE", + messageParameters = Map( + "paramIndex" -> ordinalNumber(keyIndex * 2), + "requiredType" -> toSQLType(StringType), + "inputSql" -> toSQLExpr(k), + "inputType" -> toSQLType(k.dataType))) + }.getOrElse(TypeCheckResult.TypeCheckSuccess) + if (keyCheck.isFailure) { + keyCheck + } else { + // Every value must be serializable to JSON. This mirrors `to_json`'s analysis-time + // `JacksonUtils.verifyType` check, plus a guard for the spatial atomics that check accepts + // but `JacksonGenerator` cannot write (rejecting a runtime failure at analysis instead). + memberArray.iterator.map(_._2).zipWithIndex + .foldLeft(TypeCheckResult.TypeCheckSuccess: TypeCheckResult) { + case (acc, _) if acc.isFailure => acc + case (_, (v, valueIndex)) if rawJsonArray(valueIndex) && + !(v.dataType.isInstanceOf[StringType] || v.dataType == NullType) => + // `eval` casts a raw-spliced value to UTF8String. The parser only marks a nested + // constructor (STRING-typed) raw, so reject a non-string raw value from direct + // Catalyst construction here instead of failing with a ClassCastException at eval. + DataTypeMismatch( + errorSubClass = "UNEXPECTED_INPUT_TYPE", + messageParameters = Map( + "paramIndex" -> ordinalNumber(valueIndex * 2 + 1), + "requiredType" -> toSQLType(StringType), + "inputSql" -> toSQLExpr(v), + "inputType" -> toSQLType(v.dataType))) + case (_, (v, _)) => + val elemCheck = JacksonUtils.verifyType(prettyName, v.dataType) + if (elemCheck.isFailure) { + elemCheck + } else if (JsonObjectExpr.containsUnsupportedJsonType(v.dataType)) { + DataTypeMismatch( + errorSubClass = "CANNOT_CONVERT_TO_JSON", + messageParameters = Map( + "name" -> toSQLId(prettyName), + "type" -> toSQLType(v.dataType))) + } else { + TypeCheckResult.TypeCheckSuccess + } + } + } + } + } + + override def stateful: Boolean = true + + @transient private lazy val resolvedZoneId: String = + timeZoneId.getOrElse(SQLConf.get.sessionLocalTimeZone) + + // JacksonGenerator (shared with `to_json`, so rendering matches) can only serialize a container, + // not a bare scalar, so each key/value is wrapped in a one-element array whose brackets are + // stripped in renderKey/appendRenderedValue below. + @transient private lazy val keyEvaluator: StructsToJsonEvaluator = + StructsToJsonEvaluator(Map.empty, ArrayType(StringType), Some(resolvedZoneId)) + + @transient private lazy val valueEvaluators: Array[StructsToJsonEvaluator] = + memberArray.map { case (_, v) => + StructsToJsonEvaluator(Map.empty, ArrayType(v.dataType), + Some(resolvedZoneId)) + } + + @transient private lazy val singleElem: Array[Any] = new Array[Any](1) + + @transient private lazy val singleElemData: GenericArrayData = + new GenericArrayData(singleElem) + + @transient private lazy val castInput: GenericInternalRow = + new GenericInternalRow(1) + + @transient private lazy val returningCast: Expression = + Cast(BoundReference(0, StringType, nullable = true), returning, timeZoneId, + EvalMode.ANSI) + + // `evaluateString` returns the Java String directly, skipping `evaluate`'s UTF8String round trip. + // The key is cached or appended whole, so it still materializes the interior substring. + private def renderKey(key: Any): String = { + singleElem(0) = key + val arrJson = keyEvaluator.evaluateString(singleElemData) + arrJson.substring(1, arrJson.length - 1) + } + + // Append the interior of the "[<frag>]" wrapper straight into the builder (no per-value + // substring), mirroring `JsonArray.appendRenderedElement`. + private def appendRenderedValue(sb: java.lang.StringBuilder, idx: Int, value: Any): Unit = { + singleElem(0) = value + val arrJson = valueEvaluators(idx).evaluateString(singleElemData) + sb.append(arrJson, 1, arrJson.length - 1) + } + + // A foldable, non-null key (the common literal-key case, e.g. `JSON_OBJECT('id' VALUE col)`) + // renders to the same JSON member name on every row, so render it once here instead of re-running + // the Jackson writer per row. `renderedKeys(i)` holds that cached name, or `null` when the key + // must be evaluated and rendered per row -- a non-foldable key, or a foldable key that evaluates + // to `null` (still reported as JSON_OBJECT_NULL_KEY at eval). + @transient private lazy val renderedKeys: Array[String] = memberArray.map { case (k, _) => + if (k.foldable) { + val key = k.eval(EmptyRow) + if (key == null) null else renderKey(key) + } else { + null + } + } + + override def eval(input: InternalRow): Any = { + val sb = new java.lang.StringBuilder("{") + var first = true + var i = 0 + val localMembers = memberArray + val localRawJson = rawJsonArray + while (i < localMembers.length) { + val (keyExpr, valueExpr) = localMembers(i) + // Resolve the member's key. A foldable non-null key is already rendered and cached. Otherwise + // evaluate the key before the value so a null key deterministically raises + // JSON_OBJECT_NULL_KEY, independent of whether the value expression happens to throw. + // Delay rendering dynamic keys until we know the member will be emitted, avoiding work for + // null values under ABSENT ON NULL. + val cachedKey = renderedKeys(i) + val key = if (cachedKey == null) { + val key = keyExpr.eval(input) + if (key == null) { + throw QueryExecutionErrors.jsonObjectNullKeyError() + } + key + } else { + null + } + val value = valueExpr.eval(input) + if (value != null || nullBehavior == JsonConstructorNullBehavior.Null) { + if (!first) sb.append(",") + first = false + val keyName = if (cachedKey != null) cachedKey else renderKey(key) + sb.append(keyName).append(":") + if (value == null) { + sb.append("null") + } else if (localRawJson(i)) { + sb.append(value.asInstanceOf[UTF8String].toString) + } else { + appendRenderedValue(sb, i, value) + } + } + i += 1 + } + sb.append("}") + val jsonStr = UTF8String.fromString(sb.toString) + if (returning == StringType) { + jsonStr + } else { + castInput.update(0, jsonStr) + returningCast.eval(castInput) + } + } + + override def prettyName: String = "json_object" + + override def sql: String = { + val membersSQL = members.zip(rawJson).map { case ((k, v), raw) => + // JSON_OBJECT has no value-level FORMAT JSON marker, so rawness can only be expressed in SQL + // by rendering the value as a bare JSON constructor (which reparse re-derives as implicit + // FORMAT JSON). Cover both directions of optimizer rewrites around the frozen `raw` flag: + val valueSQL = (raw, v) match { + // Raw value: a nested constructor, possibly behind collation-only wrappers (an explicit + // COLLATE or a default-collation Cast). Neither affects the raw-spliced bytes, so render + // the bare constructor and reparse re-derives raw splicing. + // TODO(SPARK-59243): a foldable nested value (e.g. JSON_ARRAY(1)) can be constant-folded to + // a string literal; with no value-level FORMAT JSON to restore rawness, this falls back to + // `v.sql` and the emitted SQL reparses quoted. Eval is unaffected (rawJson still splices). + case (true, _) => JsonObjectExpr.rawJsonConstructor(v).map(_.sql).getOrElse(v.sql) + // Quoted value the optimizer inlined here may now be a raw JSON constructor, bare or behind + // a pass-through COLLATE; neutralize it with CAST(... AS STRING) so reparse keeps it quoted + // (otherwise {"a":"{...}"} would flip to {"a":{...}}). A Cast-wrapped value already + // reparses as quoted, so it needs no extra cancellation. + case (false, _) if JsonObjectExpr.rawJsonValue(v).isDefined => s"CAST(${v.sql} AS STRING)" + case (false, _) => v.sql + } + s"${k.sql} VALUE $valueSQL" + }.mkString(", ") + val nullSQL = if (nullBehavior == JsonConstructorNullBehavior.Null) "" + else " ABSENT ON NULL" + // Use reference identity, not value equality: an explicit `RETURNING STRING COLLATE ...` + // produces a distinct StringType instance that `==` the default companion `StringType`, so `==` + // would drop it. `eq` renders nothing for anything reaching the companion (omitted RETURNING or + // plain `RETURNING STRING` alike) and renders only the distinct instance: a collated STRING. + val returningSQL = if (returning.eq(StringType)) "" else s" RETURNING ${returning.sql}" Review Comment: **Non-blocking (P2):** Direct routing depends on the lexical presence of `ON NULL` or `RETURNING`, but canonical SQL drops explicit defaults such as `NULL ON NULL` and plain `RETURNING STRING`. With a same-named session routine, reparsing the resulting clause-free SQL can resolve to that routine instead of this built-in. Please render an unambiguous direct-path clause for every resolved `JsonObjectExpr` and cover the render/reparse case under shadowing. ########## sql/core/src/test/scala/org/apache/spark/sql/JsonObjectSuite.scala: ########## @@ -0,0 +1,778 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql + +import org.apache.spark.SparkRuntimeException +import org.apache.spark.sql.catalyst.analysis.Star +import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch +import org.apache.spark.sql.catalyst.expressions.{Cast, Collate, JsonConstructorNullBehavior, JsonObjectExpr, Literal} +import org.apache.spark.sql.catalyst.parser.ParseException +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.test.SharedSparkSession +import org.apache.spark.sql.types.{CharType, GeometryType, IntegerType, MapType, StringType, VarcharType} + +/** + * End-to-end tests for the SQL:2016 `JSON_OBJECT` constructor function. + */ +class JsonObjectSuite extends QueryTest with SharedSparkSession { + import testImplicits._ + + test("basic object from key-value pairs using VALUE keyword") { + checkAnswer( + sql("SELECT json_object('id' VALUE 7, 'name' VALUE 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("construct object using optional KEY keyword") { + checkAnswer( + sql("SELECT json_object(KEY 'id' VALUE 7, KEY 'name' VALUE 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("construct object using colon syntax") { + checkAnswer( + sql("SELECT json_object('id': 7, 'name': 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("construct object using comma-separated key-value syntax") { + checkAnswer( + sql("SELECT json_object('id', 7, 'name', 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("an odd number of arguments in the comma syntax is rejected") { + // The comma form requires paired key/value arguments; a dangling key ('name') has no value. + // JSON_OBJECT is a non-reserved keyword, so when the constructor grammar cannot match, the call + // parses as an ordinary function call and routes to the registered built-in, whose builder + // rejects the odd argument count rather than silently dropping the dangling key. + val e = intercept[AnalysisException] { + sql("SELECT json_object('id', 7, 'name')") + } + assert(e.getCondition == "WRONG_NUM_ARGS.WITHOUT_SUGGESTION") + } + + test("mixing the VALUE/colon form and the comma form is a parse error") { + // The two member-list styles are mutually exclusive grammar alternatives, so a single + // constructor cannot mix `key VALUE value` (or `key : value`) members with `key, value` ones. + Seq( + "SELECT json_object('a', 1, 'b' VALUE 2)", + "SELECT json_object('a' VALUE 1, 'b', 2)", + "SELECT json_object('a' : 1, 'b', 2)").foreach { query => + intercept[ParseException](sql(query)) + } + } + + test("construct object with NULL values (default NULL ON NULL)") { + checkAnswer( + sql("SELECT json_object('id': 7, 'v': NULL)"), + Row("""{"id":7,"v":null}""")) + } + + test("construct object with explicit NULL ON NULL") { + checkAnswer( + sql("SELECT json_object('id', 7, 'v', NULL NULL ON NULL)"), + Row("""{"id":7,"v":null}""")) + } + + test("construct object with NULL values and ABSENT ON NULL") { + checkAnswer( + sql("SELECT json_object('id': 7, 'v': NULL ABSENT ON NULL)"), + Row("""{"id":7}""")) + } + + test("construct empty object") { + checkAnswer( + sql("SELECT json_object()"), + Row("{}")) + } + + test("construct object with mixed scalar types") { + checkAnswer( + sql("""SELECT json_object('int': 42, 'str': 'hello', 'bool': true, + 'float': 3.14)"""), + Row("""{"int":42,"str":"hello","bool":true,"float":3.14}""")) + } + + test("construct object with decimal type via Jackson") { + checkAnswer( + sql("""SELECT json_object('d' VALUE CAST('123.45' AS DECIMAL(5,2)))"""), + Row("""{"d":123.45}""")) + } + + test("construct object with DATE type via Jackson") { + checkAnswer( + sql("""SELECT json_object('d' VALUE DATE'2020-01-02')"""), + Row("""{"d":"2020-01-02"}""")) + } + + test("construct object with TIMESTAMP type via Jackson") { + // Note: Jackson includes timezone offset when session timezone is set + checkAnswer( + sql("""SELECT json_object('ts' VALUE TIMESTAMP'2020-01-02 10:30:00')"""), + Row("""{"ts":"2020-01-02T10:30:00.000-08:00"}""")) + } + + test("struct value renders like to_json") { + // A struct value must render exactly like `to_json` of the equivalent member. + checkAnswer( + sql("SELECT json_object('s' VALUE named_struct('a', 1, 'b', 'x'))"), + Row("""{"s":{"a":1,"b":"x"}}""")) + checkAnswer( + sql("SELECT json_object('s' VALUE named_struct('a', 1, 'b', 'x'))"), + sql("SELECT to_json(named_struct('s', named_struct('a', 1, 'b', 'x')))")) + } + + test("array value renders like to_json") { + checkAnswer( + sql("SELECT json_object('a' VALUE array(1, 2, 3))"), + Row("""{"a":[1,2,3]}""")) + checkAnswer( + sql("SELECT json_object('a' VALUE array(1, 2, 3))"), + sql("SELECT to_json(named_struct('a', array(1, 2, 3)))")) + } + + test("map value renders like to_json") { + checkAnswer( + sql("SELECT json_object('m' VALUE map('x', 1, 'y', 2))"), + Row("""{"m":{"x":1,"y":2}}""")) + checkAnswer( + sql("SELECT json_object('m' VALUE map('x', 1, 'y', 2))"), + sql("SELECT to_json(named_struct('m', map('x', 1, 'y', 2)))")) + } + + test("nested complex value combining struct, array and map renders like to_json") { + val value = "named_struct('arr', array(1, 2), 'm', map('k', named_struct('n', 3)))" + checkAnswer( + sql(s"SELECT json_object('c' VALUE $value)"), + sql(s"SELECT to_json(named_struct('c', $value))")) + } + + test("struct value honors spark.sql.jsonGenerator.ignoreNullFields like to_json") { + // `ON NULL` controls only top-level members; a null field *inside* a struct value follows + // spark.sql.jsonGenerator.ignoreNullFields, like `to_json`. + val value = "named_struct('a', 1, 'b', CAST(NULL AS INT))" + Seq("true", "false").foreach { ignore => + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> ignore) { + checkAnswer( + sql(s"SELECT json_object('s' VALUE $value)"), + sql(s"SELECT to_json(named_struct('s', $value))")) + } + } + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> "false") { + checkAnswer(sql(s"SELECT json_object('s' VALUE $value)"), Row("""{"s":{"a":1,"b":null}}""")) + } + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> "true") { + checkAnswer(sql(s"SELECT json_object('s' VALUE $value)"), Row("""{"s":{"a":1}}""")) + } + } + + test("top-level ON NULL and struct-internal ignoreNullFields are independent") { + // With NULL ON NULL (default) and ignoreNullFields=true, a top-level NULL member is kept as + // `null` while a null field inside a struct value is dropped. + withSQLConf(SQLConf.JSON_GENERATOR_IGNORE_NULL_FIELDS.key -> "true") { + checkAnswer( + sql("""SELECT json_object('top' VALUE CAST(NULL AS INT), + 's' VALUE named_struct('a', 1, 'b', CAST(NULL AS INT)))"""), + Row("""{"top":null,"s":{"a":1}}""")) + } + } + + test("string escaping in keys") { + checkAnswer( + sql("""SELECT json_object('key"with"quotes' VALUE 1)"""), + Row("""{"key\"with\"quotes":1}""")) + } + + // For scalar string values JSON_OBJECT must escape exactly like to_json of the equivalent + // struct (both go through the same Jackson generator); assert that equivalence rather than + // hand-encoding the escaping, which is easy to get wrong across Scala/SQL/JSON layers. + test("string escaping in values matches to_json") { + checkAnswer( + sql("""SELECT json_object('msg' VALUE 'hello +world')"""), + sql("""SELECT to_json(named_struct('msg', 'hello +world'))""")) + } + + test("string escaping with backslash matches to_json") { + checkAnswer( + sql("""SELECT json_object('path' VALUE 'c:\windows')"""), + sql("""SELECT to_json(named_struct('path', 'c:\windows'))""")) + } + + test("nested JSON_OBJECT spliced raw") { + checkAnswer( + sql("""SELECT json_object('a' VALUE json_object('b' VALUE 1))"""), + Row("""{"a":{"b":1}}""")) + checkAnswer( + sql("""SELECT json_object('a', json_object('b', 1))"""), + Row("""{"a":{"b":1}}""")) + } + + test("nested JSON_OBJECT with multiple levels") { + checkAnswer( + sql("""SELECT json_object('outer' VALUE + json_object('inner' VALUE 42, 'name' VALUE 'test'))"""), + Row("""{"outer":{"inner":42,"name":"test"}}""")) + } + + test("a nested JSON_ARRAY value is spliced raw") { + checkAnswer( + sql("SELECT json_object('a' VALUE json_array(1, 2))"), + Row("""{"a":[1,2]}""")) + } + + test("JSON_OBJECT nested directly in JSON_ARRAY is spliced as an object element") { + // The inverse nesting direction: a JSON_OBJECT in a JSON_ARRAY element position stays on the + // direct grammar path (JsonArrayValueContext), so it is spliced as a JSON object rather than + // routed through resolution and emitted as a quoted string. + checkAnswer( + sql("SELECT json_array(json_object('a', 1), json_object('b', 2))"), + Row("""[{"a":1},{"b":2}]""")) + } + + test("a nested JSON_QUERY value is spliced under KEEP QUOTES and quoted under OMIT QUOTES") { + // JSON_QUERY emits JSON text under the default KEEP QUOTES, so a lexically nested JSON_QUERY is + // spliced raw: the matched object is {"x":1}, not the quoted string "{\"x\":1}". + checkAnswer( + sql("""SELECT json_object('a' VALUE json_query('{"o":{"x":1}}', '$.o'))"""), + Row("""{"a":{"x":1}}""")) + // OMIT QUOTES returns the matched scalar string's decoded content (Ada, not "Ada") -- an + // ordinary string -- so it takes the quoted path (emitsImplicitJsonText is false), never the + // invalid splice {"a":Ada}. + checkAnswer( + sql("""SELECT json_object('a' VALUE json_query('{"n":"Ada"}', '$.n' OMIT QUOTES))"""), + Row("""{"a":"Ada"}""")) + } + + test("null key error") { + val e = intercept[SparkRuntimeException] { + sql("SELECT json_object(NULL VALUE 'value')").collect() + } + // Assert the structured error contract, not just the message text. + assert(e.getCondition == "JSON_OBJECT_NULL_KEY") + assert(e.getSqlState == "2200E") + } + + test("a null key is validated before a null value is omitted under ABSENT ON NULL") { + // ABSENT ON NULL omits members with a null value, but the key is validated first, so a null key + // still raises JSON_OBJECT_NULL_KEY rather than being silently dropped along with the member. + val e = intercept[SparkRuntimeException] { + sql("SELECT json_object(NULL VALUE NULL ABSENT ON NULL)").collect() + } + assert(e.getCondition == "JSON_OBJECT_NULL_KEY") + assert(e.getSqlState == "2200E") + } + + test("non-foldable key and value expressions") { + val df = Seq(("key1", "val1"), ("key2", "val2")).toDF("k", "v") + checkAnswer( + df.selectExpr("json_object(k VALUE v)"), + Seq(Row("""{"key1":"val1"}"""), Row("""{"key2":"val2"}"""))) + } + + test("non-foldable with NULL value and NULL ON NULL") { + val df = Seq(("k", null), ("key", "val")).toDF("k", "v") + checkAnswer( + df.selectExpr("json_object(k VALUE v)"), + Seq(Row("""{"k":null}"""), Row("""{"key":"val"}"""))) + } + + test("non-foldable with NULL value and ABSENT ON NULL") { + val df = Seq(("k", null), ("key", "val")).toDF("k", "v") + checkAnswer( + df.selectExpr("json_object(k VALUE v ABSENT ON NULL)"), + Seq(Row("{}"), Row("""{"key":"val"}"""))) + } + + test("multiple keys with ABSENT ON NULL") { + checkAnswer( + sql("""SELECT json_object('a' VALUE 1, 'b' VALUE NULL, 'c' VALUE 3 + ABSENT ON NULL)"""), + Row("""{"a":1,"c":3}""")) + } + + test("duplicate keys are emitted in source order") { + checkAnswer( + sql("SELECT json_object('k' VALUE 1, 'k' VALUE 2)"), + Row("""{"k":1,"k":2}""")) + } + + test("non-string key type is rejected at analysis, not at execution") { + val ex = intercept[AnalysisException] { + sql("SELECT json_object(1 VALUE 'x')") + } + assert(ex.getMessage.contains("UNEXPECTED_INPUT_TYPE")) + } + + test("non-string key type reports the actual key argument") { + val ex = intercept[AnalysisException] { + sql("SELECT json_object('ok' VALUE 1, 2 VALUE 'bad')") + } + checkError( + exception = ex, + condition = "DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE", + sqlState = Some("42K09"), + parameters = Map( + "sqlExpr" -> "\"JSON_OBJECT(ok VALUE 1, 2 VALUE bad)\"", + "paramIndex" -> "third", + "requiredType" -> "\"STRING\"", + "inputSql" -> "\"2\"", + "inputType" -> "\"INT\""), + queryContext = Array(ExpectedContext("json_object('ok' VALUE 1, 2 VALUE 'bad')", 7, 46))) + } + + test("collated STRING RETURNING is accepted") { + // isValidReturningType must accept any StringType instance, not just the default collation. + checkAnswer( + sql("SELECT json_object('a' VALUE 1 RETURNING STRING COLLATE UTF8_LCASE)"), + Row("""{"a":1}""")) + } + + test("an invalid RETURNING type is reported under DATATYPE_MISMATCH") { + // The error is emitted as a DataTypeMismatch, so its condition must resolve under + // DATATYPE_MISMATCH -- not as a top-level INVALID_JSON_RETURNING_TYPE class. + val e = intercept[AnalysisException] { + sql("SELECT json_object('a' VALUE 1 RETURNING INT)").collect() + } + assert(e.getCondition == "DATATYPE_MISMATCH.INVALID_JSON_RETURNING_TYPE") + } + + test("a directly-constructed JsonObjectExpr with a CHAR/VARCHAR RETURNING is rejected") { + // The parser normalizes CHAR/VARCHAR RETURNING to STRING, but a raw CharType/VarcharType from + // direct Catalyst construction would advertise a length JSON_OBJECT does not enforce. + Seq(VarcharType(2), CharType(2)).foreach { returning => + val expr = JsonObjectExpr( + Seq((Literal("k"), Literal(1))), Seq(false), JsonConstructorNullBehavior.Null, returning) + expr.checkInputDataTypes() match { + case DataTypeMismatch(errorSubClass, _) => + assert(errorSubClass == "INVALID_JSON_RETURNING_TYPE", s"for $returning") + case other => fail(s"expected DataTypeMismatch for $returning, got $other") + } + } + } + + test("value accepts an unparenthesized predicate expression") { + // valueExpr is parsed as a full `expression`, so ordinary predicates work without parentheses. + checkAnswer(sql("SELECT json_object('isnull' VALUE 1 IS NULL)"), Row("""{"isnull":false}""")) + checkAnswer(sql("SELECT json_object('gt' : 2 > 1)"), Row("""{"gt":true}""")) + } + + test("widening the value to expression does not change documented forms") { + // Design-doc examples where a value abuts the ON NULL / RETURNING keywords must still parse and + // evaluate identically after widening valueExpression -> expression. + checkAnswer(sql("SELECT json_object('id': 7, 'v': NULL)"), Row("""{"id":7,"v":null}""")) + checkAnswer( + sql("SELECT json_object('id': 7, 'v': NULL ABSENT ON NULL)"), Row("""{"id":7}""")) + checkAnswer( + sql("SELECT json_object('id', 7, 'v', NULL ABSENT ON NULL)"), Row("""{"id":7}""")) + checkAnswer( + sql("SELECT json_object('id' VALUE 7, 'name' VALUE 'Ada')"), + Row("""{"id":7,"name":"Ada"}""")) + } + + test("an unsupported value type is rejected at analysis") { + // A spatial value: JacksonUtils.verifyType accepts it (it is an AtomicType) but + // JacksonGenerator cannot serialize it, so JSON_OBJECT must reject it up front, not at runtime. + val bad = JsonObjectExpr( + Seq((Literal("k"), Literal.create(null, GeometryType(4326)))), + Seq(false), JsonConstructorNullBehavior.Null, StringType) + bad.checkInputDataTypes() match { + case DataTypeMismatch(sub, _) => assert(sub == "CANNOT_CONVERT_TO_JSON") + case other => fail(s"expected DataTypeMismatch, got $other") + } + // A spatial type appearing only as a MAP KEY is fine: JacksonGenerator writes map keys via + // toString, so the value-type guard must not over-reject it. + val ok = JsonObjectExpr( + Seq((Literal("k"), Literal.create(null, MapType(GeometryType(4326), IntegerType)))), + Seq(false), JsonConstructorNullBehavior.Null, StringType) + assert(ok.checkInputDataTypes().isSuccess) + } + + test("a directly-constructed raw value that is not a string is rejected") { + // The parser only marks a nested constructor (STRING-typed) raw; a non-string raw value from + // direct construction would fail with a ClassCastException at eval, so reject it at analysis. + val expr = JsonObjectExpr( + Seq((Literal("k"), Literal(1))), Seq(true), JsonConstructorNullBehavior.Null, StringType) + expr.checkInputDataTypes() match { + case DataTypeMismatch(sub, _) => assert(sub == "UNEXPECTED_INPUT_TYPE") + case other => fail(s"expected DataTypeMismatch, got $other") + } + } + + test("SQL renders an explicit collated RETURNING and omits only the default") { + val collated = JsonObjectExpr( + Seq((Literal("k"), Literal(1))), Seq(false), JsonConstructorNullBehavior.Null, + StringType("UTF8_LCASE")) + assert(collated.sql.contains("RETURNING STRING COLLATE UTF8_LCASE")) + // The omitted default is the companion StringType (by reference) and renders no RETURNING. + val default = JsonObjectExpr( + Seq((Literal("k"), Literal(1))), Seq(false), JsonConstructorNullBehavior.Null, StringType) + assert(default.sql == "JSON_OBJECT('k' VALUE 1)") + } + + test("SQL renders a raw nested value as a bare constructor even after collation wrapping") { + val inner = JsonObjectExpr( + Seq((Literal("b"), Literal(1))), Seq(false), JsonConstructorNullBehavior.Null, StringType) + // Simulate the default-collation rule wrapping the raw nested value in a Cast. rawJson stays + // frozen true; .sql must render the bare constructor so reparse re-derives raw splicing (there + // is no value-level FORMAT JSON marker in JSON_OBJECT). + val wrapped = JsonObjectExpr( + Seq((Literal("a"), Cast(inner, StringType("UTF8_LCASE")))), Seq(true), + JsonConstructorNullBehavior.Null, StringType) + assert(wrapped.sql == "JSON_OBJECT('a' VALUE JSON_OBJECT('b' VALUE 1))") + } + + test("emitted SQL reparses and evaluates with raw-vs-quoted semantics preserved") { + // The .sql renderings above are round-trip contracts: reparsing and evaluating them must + // reproduce the original raw-vs-quoted splicing. + // A raw nested value renders as a bare constructor and reparses back to raw splicing. + checkAnswer( + sql("SELECT JSON_OBJECT('a' VALUE JSON_OBJECT('b' VALUE 1))"), Row("""{"a":{"b":1}}""")) + // A quoted value that the optimizer inlined as an implicit-JSON expression is neutralized with + // CAST(... AS STRING); reparsing must keep it quoted rather than splicing it raw. + val inner = JsonObjectExpr( + Seq((Literal("b"), Literal(1))), Seq(false), JsonConstructorNullBehavior.Null, StringType) + val quoted = JsonObjectExpr( + Seq((Literal("a"), inner)), Seq(false), JsonConstructorNullBehavior.Null, StringType) + assert(quoted.sql == "JSON_OBJECT('a' VALUE CAST(JSON_OBJECT('b' VALUE 1) AS STRING))") + checkAnswer(sql(s"SELECT ${quoted.sql}"), Row("""{"a":"{\"b\":1}"}""")) + } + + test("JSON_OBJECT is not foldable") { + // Folding a constant JSON_OBJECT would (a) surface a null-key error at optimization even for + // rows a filter/join drops, and (b) fold a nested raw JSON_OBJECT value to a string literal, + // which .sql could no longer render as a bare constructor (JSON_OBJECT has no value-level + // FORMAT JSON marker). So it stays non-foldable. + assert(!JsonObjectExpr( + Seq((Literal("k"), Literal(1))), Seq(false), + JsonConstructorNullBehavior.Null, StringType).foldable) + } + + test("a null key raises JSON_OBJECT_NULL_KEY before the value is evaluated") { + // The key is checked before the value is evaluated, so a null key wins deterministically even + // when the value expression would itself throw. + // `raise_error(k)` references the column so it is neither foldable nor evaluated before the + // key null-check; if the value ran first the error would come from `raise_error`, not the key. + val e = intercept[SparkRuntimeException] { + sql("SELECT json_object(k VALUE raise_error(k)) " + + "FROM VALUES (CAST(NULL AS STRING)) t(k)").collect() + } + assert(e.getCondition == "JSON_OBJECT_NULL_KEY") + assert(e.getSqlState == "2200E") + } + + test("a foldable literal key is rendered once and reused across rows") { + // JSON_OBJECT caches the rendered name of a foldable non-null key; the same key must still be + // emitted for every row. + checkAnswer( + sql("SELECT json_object('id' VALUE a) FROM VALUES (1), (2) t(a)"), + Seq(Row("""{"id":1}"""), Row("""{"id":2}"""))) + // A foldable key that evaluates to null is not cached: it must still raise JSON_OBJECT_NULL_KEY + // per row rather than being silently skipped. + Seq("NULL", "CAST(NULL AS STRING)").foreach { k => + val e = intercept[SparkRuntimeException] { + sql(s"SELECT json_object($k VALUE 1)").collect() + } + assert(e.getCondition == "JSON_OBJECT_NULL_KEY", s"for key $k") + assert(e.getSqlState == "2200E", s"for key $k") + } + } + + test("CHAR/VARCHAR RETURNING is normalized to STRING regardless of preserveCharVarcharTypeInfo") { + Seq("CHAR(2)", "VARCHAR(2)").foreach { returning => + Seq("true", "false").foreach { preserve => + withSQLConf(SQLConf.PRESERVE_CHAR_VARCHAR_TYPE_INFO.key -> preserve) { + assert( + sql(s"SELECT json_object('k' VALUE 1 RETURNING $returning)").schema.head.dataType + === StringType, + s"for RETURNING $returning, preserveCharVarcharTypeInfo=$preserve") + } + } + } + } + + test("object default collation applies only when RETURNING is not explicitly collated") { + withSQLConf(SQLConf.OBJECT_LEVEL_COLLATIONS_ENABLED.key -> "true") { + withTable("t") { + sql( + """CREATE TABLE t DEFAULT COLLATION UTF8_LCASE AS + |SELECT json_object('k' VALUE 1) AS a, + | json_object('k' VALUE 1 RETURNING STRING COLLATE UTF8_BINARY) AS b""".stripMargin) + val schema = spark.table("t").schema + // Omitted RETURNING (default STRING) follows the table's default collation. + assert(schema("a").dataType === StringType("UTF8_LCASE")) + // Explicit RETURNING ... COLLATE is the user's choice and must not be overwritten. + assert(schema("b").dataType === StringType("UTF8_BINARY")) + } + } + } + + test("default collation recurses into a nested JSON_OBJECT value") { + // The rule casts each DefaultStringProducingExpression, recursing through a nested constructor + // (the flat cases above only cover a top-level constructor). This CTAS runs the default + // analyzer (single-pass included). Confirm the schema collation and that raw splicing still + // produces well-formed nested JSON at runtime. + withSQLConf(SQLConf.OBJECT_LEVEL_COLLATIONS_ENABLED.key -> "true") { + withTable("t") { + sql( + """CREATE TABLE t DEFAULT COLLATION UTF8_LCASE AS + |SELECT json_object('a' VALUE json_object('b' VALUE 1)) AS a""".stripMargin) + assert(spark.table("t").schema("a").dataType === StringType("UTF8_LCASE")) + checkAnswer(spark.table("t"), Row("""{"a":{"b":1}}""")) + } + } + } + + test("view default collation preserves an explicit collated RETURNING") { + // Exercises the CREATE VIEW resolution path (in addition to the CTAS path above): the explicit + // RETURNING collation must survive the view's default collation. Pin the fixed-point analyzer: + // the single-pass resolver does not yet resolve a TimeZoneAware JSON constructor's timezone + // when re-resolving a view (a pre-existing gap independent of collation); the CTAS test above + // already exercises the single-pass path via the dual-run analyzer. + withSQLConf( + SQLConf.ANALYZER_DUAL_RUN_LEGACY_AND_SINGLE_PASS_RESOLVER.key -> "false", Review Comment: **Non-blocking (P2):** This opt-out hides a supported-path mismatch: `JsonObjectExpr` is admitted by `ResolverGuard`, but view re-resolution can reach it without the timezone required for resolution. Please route the view-specific handoff through the shared timezone-aware expression resolution, then keep this case under dual-run parity and verify default and explicitly collated RETURNING types. **Recommended change:** Ensure re-resolved view expressions admitted by ResolverGuard receive the same timezone completion and default-collation coercion as ordinary single-pass TimeZoneAwareExpression resolution, then remove the JSON_OBJECT dual-run opt-out. **Why this works:** Trace the view-specific expression handoff and route a timezone-empty JsonObjectExpr through the shared TimezoneAwareExpressionResolver before resolved-expression validation, preserving the existing collation cast behavior rather than adding a JSON_OBJECT-only evaluation fallback. **Scope:** Restore analyzer parity at the shared view/timezone resolution boundary and make JSON_OBJECT view coverage enforce it. **Compatibility:** Fixed-point analysis results, explicit JSON_OBJECT RETURNING collations, and existing JSON_ARRAY view semantics must remain unchanged. **Risks:** Reapplying timezone or coercion to already-resolved view subtrees can duplicate casts or change tags unless the existing resolver helpers remain authoritative. A broad view-path change can affect other TimeZoneAwareExpression implementations, so parity coverage must include representative existing behavior. **Constraints:** Preserve explicit RETURNING collations and view default-collation behavior. Do not populate timezone only lazily at evaluation time; the resolved-plan invariant must hold. **Success:** A JSON_OBJECT in a CREATE VIEW expression has a non-empty timezone in both analyzers. The view scenario runs with dual-run parity enabled and preserves omitted/default and explicitly collated RETURNING types. Existing JSON_ARRAY view resolution remains unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
