srielau opened a new pull request, #59453: URL: https://github.com/apache/spark/pull/59453
### What changes were proposed in this pull request? Follow-up to SPARK-60108. 1. `OptimizeCsvJsonExprs` no longer folds `from_json(to_json(m))` to `m` when `JsonToStructs.charVarcharStandardSemantics` is on and the schema has a CHAR/VARCHAR map key at any nesting. That fold skipped the JSON map-key length check when `m` already carried an off-width key. 2. Nested `JacksonParser` converters that fail with `spark.sql.json.enablePartialResults=false` now drain their object or array through `END_OBJECT` / `END_ARRAY` before rethrowing, so inner `FIELD_NAME`s cannot be taken as parent siblings. This is a parser-position fix for any nested struct, map, or array `NonFatal` failure, not only CHAR/VARCHAR map keys. Skip failures are suppressed on the original cause. STRING-key round-trips and the flag-off path still fold. ### Why are the changes needed? SPARK-60108 length-checks JSON CHAR/VARCHAR map keys without pad or trim. The optimizer identity fold treated `from_json(to_json(m))` as a no-op, so a map that already had an off-width key (for example from a view created with the flag off) never ran that check. Separately, a nested converter that failed non-partially left Jackson mid-container. The parent field loop could then assign inner names to outer fields (or overwrite fields already parsed). CHAR/VARCHAR map-key rejection was the motivating case; any nested struct/map/array failure had the same hole. JIRA: https://issues.apache.org/jira/browse/SPARK-60229 ### Does this PR introduce _any_ user-facing change? Yes, on unreleased master / branch-4.x (SPARK-60108 is not in a released Spark version). - `from_json(to_json(m))` with CHAR/VARCHAR map keys under standard semantics now applies the key length check instead of returning `m` unchanged. - Nested JSON parse failures with `spark.sql.json.enablePartialResults=false` (internal, default true) now keep already-parsed siblings and continue after the failed nested value instead of stealing inner field names. Default `true` already contained most nested failures; this closes the non-partial nested path, including FAILFAST still reporting the original error. Example (standard semantics on): `from_json(to_json(m), 'MAP<CHAR(3), INT>')` for `m` with key `a` is null in PERMISSIVE and `UNSUPPORTED_JSON_CHAR_VARCHAR_MAP_KEY` in FAILFAST, rather than returning `m`. ### How was this patch tested? - `OptimizeJsonExprsSuite`: fold skipped for CHAR/VARCHAR/nested/array map keys with the flag on; STRING keys and flag-off still fold. - `BasicCharVarcharTestSuite`: `from_json(to_json)` for CHAR and VARCHAR; nested struct steal both field orders; array element failure; FAILFAST still the map-key error. - Existing SPARK-40646 / SPARK-44940 / SPARK-59274 expectations updated so siblings after a nested failure are parsed when partial results are off. ``` build/sbt 'catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.OptimizeJsonExprsSuite -- -z SPARK-60229' build/sbt 'sql/testOnly org.apache.spark.sql.BasicCharVarcharTestSuite -- -z SPARK-60229' build/sbt 'sql/testOnly org.apache.spark.sql.JsonFunctionsSuite -- -z SPARK-40646' build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.json.JsonV1Suite -- -z SPARK-40646 -z SPARK-44940' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
