srielau opened a new pull request, #58936: URL: https://github.com/apache/spark/pull/58936
### What changes were proposed in this pull request? Support first-class CHAR/VARCHAR output from script TRANSFORM when first-class semantics are enabled. Split out from [#58584](https://github.com/apache/spark/pull/58584) (Hive inspector conversion + UDF/UDAF/UDTF) per review feedback. - **No-SerDe path:** CHAR/VARCHAR scalar, nested, and complex output with padding, overflow (`EXCEED_LIMIT_LENGTH`), null-token, and malformed-field handling. - **No-SerDe nested maps:** Restore JSON string keys to the declared physical key type through a per-row `ArrayBasedMapBuilder` that validates null and duplicate converted keys. - **SerDe path:** Rewrite `LazySimpleSerDe` CHAR/VARCHAR output types to STRING so Spark applies first-class length checks. Non-`LazySimpleSerDe` SerDes keep the declared CHAR/VARCHAR schema. - **Legacy:** Legacy flag-off behavior remains unchanged. This PR depends on [#58584](https://github.com/apache/spark/pull/58584) for Hive inspector CHAR/VARCHAR conversion. A JIRA subtask under SPARK-59277 will be created for this work. ### Why are the changes needed? With `spark.sql.charVarchar.standardSemantics.enabled=true`, script TRANSFORM output was not aware of CHAR/VARCHAR types. Bounded-string values passed through without padding or length enforcement, and nested output containing non-string map keys was not restored from the JSON string representation. ### Does this PR introduce _any_ user-facing change? Yes. When first-class CHAR/VARCHAR semantics are enabled, script TRANSFORM now pads CHAR output, enforces length limits on both CHAR and VARCHAR, and correctly handles nested CHAR/VARCHAR in arrays, maps, and structs. Legacy flag-off behavior remains unchanged. ### How was this patch tested? Added focused coverage for: - Scalar CHAR/VARCHAR output without SerDe (padding, overflow). - Nested CHAR/VARCHAR (arrays, maps with non-string keys, structs) without SerDe. - Malformed JSON and colliding map-key restoration without SerDe. - Colliding row followed by valid row in the same partition (builder-leak regression). - Nested collated CHAR/VARCHAR with Hive SerDe. - Scalar CHAR/VARCHAR overflow with Hive SerDe. - `LazySimpleSerDe`-only CHAR/VARCHAR type rewrite verification. Ran: ``` sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ -Dsbt.override.build.repos=true \ 'sql/testOnly org.apache.spark.sql.execution.SparkScriptTransformationSuite -- -z SPARK-59277' sbt -java-home /usr/lib/jvm/java-17-openjdk-amd64 \ -Dsbt.override.build.repos=true \ 'hive/testOnly org.apache.spark.sql.hive.execution.HiveScriptTransformationSuite -- -z SPARK-59277' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Auto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
