ganeshashree commented on code in PR #58450:
URL: https://github.com/apache/spark/pull/58450#discussion_r3946346611
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala:
##########
@@ -4267,38 +4299,42 @@ class AstBuilder extends DataTypeAstBuilder
*/
override def visitJsonArray(ctx: JsonArrayContext): Expression =
withOrigin(ctx) {
val arrayValues = ctx.values.asScala.map(v => expression(v.value)).toSeq
- // Freeze the FORMAT JSON decisions here, from the lexical argument, so a
later
- // analyzer/optimizer rewrite that wraps or swaps the child cannot change
them (see
- // [[ImplicitlyFormattedAsJson]]). For each element:
- // - `formatJson`: whether it is already-JSON text spliced raw. True when
it carries an
- // explicit `FORMAT JSON` clause, or is a (lexically) nested JSON
constructor -- seen through
- // a value-preserving `COLLATE` via `JsonArray.isImplicitlyJson`.
- // - `needsValidation`: whether its raw text is arbitrary user input to
JSON-validate at eval.
- // True only for an explicit `FORMAT JSON` on something that is NOT a
JSON constructor; a
- // nested constructor emits well-formed JSON by construction and is
trusted.
- val formatArgs = ctx.values.asScala.zip(arrayValues).map { case (v, expr)
=>
- val explicit = v.FORMAT() != null
- val implicitlyJson = JsonArray.isImplicitlyJson(expr)
- (explicit || implicitlyJson, explicit && !implicitlyJson)
- }.toSeq
- val formatJson = formatArgs.map(_._1)
- val needsValidation = formatArgs.map(_._2)
- // Default RETURNING type is STRING; the result is JSON text. A
CHAR/VARCHAR RETURNING is
- // normalized to STRING unconditionally: JSON_ARRAY serializes the
fragment itself and never
- // advertises a CHAR/VARCHAR length it does not enforce. The
CharVarcharUtils helpers cannot be
- // used here -- they honor spark.sql.preserveCharVarcharTypeInfo and would
leave a VARCHAR(n)
- // length in the output type when that flag is set. A non-string RETURNING
is left intact for
- // checkInputDataTypes to fail.
- val returning = Option(ctx.returning).map(typedVisit[DataType]).map {
- case c: CharType => c.toStringType
- case v: VarcharType => v.toStringType
- case other => other
- }.getOrElse(StringType)
- // Default ON NULL behavior is ABSENT ON NULL (drop NULL elements).
- val nullBehavior = Option(ctx.nullBehavior)
- .map(buildJsonConstructorNullBehavior)
- .getOrElse(JsonConstructorNullBehavior.Absent)
- JsonArray(arrayValues, formatJson, needsValidation, nullBehavior,
returning)
+ val hasExplicitFormat = ctx.values.asScala.exists(_.FORMAT() != null)
+ val hasImplicitJson = arrayValues.exists(JsonArray.isImplicitlyJson)
+ if (!hasExplicitFormat && ctx.nullBehavior == null && ctx.returning ==
null &&
+ !isTopLevelJsonArrayElement(ctx) && !hasImplicitJson) {
Review Comment:
You're right, this is a real inconsistency. With a shadow-first PATH,
`json_array(1)` routes and gets shadowed, but `json_array(json_array(1))` takes
the direct-builtin branch via `hasImplicitJson`, so the built-in runs and the
shadow is skipped.
The catch is that I can't just flip these calls to route without first
fixing a limitation that already exists in `json_array` itself. When the
built-in is reached through a routed or qualified call, it can't splice a
nested constructor today, it quotes it (that's the `["[1]"]` vs `[[1]]` gap I
flagged as the SPARK-59243 TODO in `JsonArrayExpressionBuilder`). Right now
that limitation stays hidden because a nested clause-free call never routes: it
goes down the direct grammar branch, which splices correctly. The moment I
route it to make shadowing work, I hit that gap and the built-in owner produces
the wrong output.
I would keep this PR to flat clause-free calls and finish the nested case in
SPARK-59243. Happy to pull it in here if you'd prefer, but it drags that whole
change in with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
