srielau opened a new pull request, #59302: URL: https://github.com/apache/spark/pull/59302
### What changes were proposed in this pull request? Keep Hive `TRANSFORM` / `MAP` / `REDUCE` script I/O as unbounded `STRING` (JSON for complex types), the same as sources that have no native CHAR/VARCHAR. After the script reader, apply Spark CHAR/VARCHAR assignment (`CharVarcharUtils.stringLengthCheck`: pad `CHAR`, `EXCEED_LIMIT_LENGTH` on overflow) in a `Project` so the query result type is the declared `CHAR`/`VARCHAR` (including nested types). This supersedes the SPARK-59683 codec inside the script/Hive path: - Recursively replace CHAR/VARCHAR with unbounded STRING for script exec output (including nested maps/arrays/structs). - Parse delimited / JSON as today for STRING-shaped types. - When `spark.sql.charVarchar.standardSemantics.enabled` is true, wrap with a Project using `stringLengthCheck`. - Flag off: keep today's STRING result (`AS (c CHAR(5))` remains STRING). Preserve-only (`preserveCharVarcharTypeInfo` without standard semantics): no pad / no `EXCEED_LIMIT_LENGTH`. - Remove CHAR-specific logic from no-SerDe converters, Hive unwrappers, SerDe schema rewrite, and the bind-at-parse I/O-schema flag. JSON non-string map keys (`MAP<INT, ...>`, `MAP<DATE, ...>`) remain out of scope. JIRA: https://issues.apache.org/jira/browse/SPARK-60090 Parent: SPARK-59683 / PR #58936 ### Why are the changes needed? SPARK-59683 mixed CHAR/VARCHAR assignment (pad / length, same as CSV/JSON/Parquet) with TRANSFORM JSON / Hive wire issues that are not CHAR-specific. That made `MAP<STRING, STRING>` vs `MAP<STRING, CHAR(2)>` diverge. TRANSFORM is Hive-compat STRING-on-the-wire; assignment belongs in a Spark `Project` like any other STRING source. ### Does this PR introduce _any_ user-facing change? Yes, only when `spark.sql.charVarchar.standardSemantics.enabled` is true (unreleased path): - `SELECT TRANSFORM(...) USING 'cat' AS (c CHAR(4))` returns typed `CHAR(4)` with padding. - Overflow raises `EXCEED_LIMIT_LENGTH`. - Nested `ARRAY` / `STRUCT` / `MAP<CHAR, VARCHAR>` go through the same Project assignment. Flag off is unchanged: `AS (c CHAR(5))` remains STRING with no pad and no length check. ### How was this patch tested? - `sql/testOnly org.apache.spark.sql.execution.SparkScriptTransformationSuite -- -z SPARK-60090` (9 passed) - `hive/testOnly org.apache.spark.sql.hive.execution.HiveScriptTransformationSuite -- -z SPARK-60090` (8 passed) - `catalyst/testOnly org.apache.spark.sql.catalyst.parser.PlanParserSuite -- -z TRANSFORM` Coverage includes scalar pad/overflow/null token, nested ARRAY/STRUCT/MAP assignment, flag-off and preserve-only STRING results, Hive SerDe going through the same Spark-side Project, and views keeping the parsed Project. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
