srielau opened a new pull request, #58545: URL: https://github.com/apache/spark/pull/58545
### What changes were proposed in this pull request? When `spark.sql.charVarchar.standardSemantics.enabled` is true, first-class CHAR/VARCHAR schemas are already allowed through `failIfHasCharVarchar`. Schema-driven text parsers still treated those columns as unbounded STRING (no pad/overflow), `schema_of_json/csv/xml` rejected CHAR/VARCHAR input with `child.dataType != StringType`, and XML map keys / `convertTo` used exact `StringType` matches. This patch applies assignment semantics while parsing text into a typed schema: - JSON (`JacksonParser`), CSV (`UnivocityParser`), and XML (`StaxXmlParser`) call `CharVarcharUtils.applyTextParseSemantics` so CHAR is padded and oversize VARCHAR raises `EXCEED_LIMIT_LENGTH` - JSON/XML map keys with CHAR/VARCHAR key types get the same checks - XML wildcard columns and `convertTo` keep the declared CHAR/VARCHAR type instead of collapsing to `StringType` - `schema_of_json` / `schema_of_csv` / `schema_of_xml` accept any `StringType` subtype as the input document Default (flag off) is unchanged: `from_json(..., 'a CHAR(5)')` still fails with `UNSUPPORTED_CHAR_OR_VARCHAR_AS_STRING`. ### Why are the changes needed? `from_json` / `from_csv` / `from_xml` are the remaining schema-driven text parse surfaces that drop CHAR/VARCHAR length rules under standard semantics. Without this, enabling the flag still produces unpadded CHAR values, silently truncates or accepts oversize VARCHAR, and rejects CHAR/VARCHAR documents in `schema_of_*`. JIRA: https://issues.apache.org/jira/browse/SPARK-59274 (subtask of SPARK-58794) ### Does this PR introduce _any_ user-facing change? Yes, when `spark.sql.charVarchar.standardSemantics.enabled` is true (still default false): - `from_json` / `from_csv` / `from_xml` with a CHAR/VARCHAR schema keep those types and apply assignment checks (CHAR pad, VARCHAR overflow -> `EXCEED_LIMIT_LENGTH`). - `schema_of_json` / `schema_of_csv` / `schema_of_xml` accept CHAR/VARCHAR input strings. - JSON/XML maps with CHAR/VARCHAR keys pad or reject keys the same way. ### How was this patch tested? Added `BasicCharVarcharTestSuite` coverage for SPARK-59274: CHAR padding and VARCHAR overflow in `from_json` / `from_csv` / `from_xml`, JSON/XML `MAP<CHAR(n), INT>` keys, and `schema_of_json/csv/xml` on `VARCHAR` input. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Cursor Grok 4.6 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
