srielau opened a new pull request, #58080: URL: https://github.com/apache/spark/pull/58080
### What changes were proposed in this pull request? **Depends on #58033** (CHAR/VARCHAR standardSemantics foundation). Please review/merge that first; rebase this onto `master` afterward. Delta vs foundation: https://github.com/srielau/spark/compare/SPARK-58794...SPARK-58794-lct-compare Follow-up that drills into least-common-type, comparison, and `IN` under `spark.sql.charVarchar.standardSemantics.enabled`. **Bug fix.** `CollationTypeCoercion.getWinningStringType` required `sameType` at equal collation strength. Same-collation `CHAR(2)` and `CHAR(4)` therefore looked like a collation mismatch and became `IndeterminateStringType` (`string collate null`). At equal strength, take `StringHelper.tightestCommonString` instead so every child is cast to `max(n, m)` (padding, never truncating). **Coverage / pinned behavior.** - Set ops: `UNION` / `UNION ALL` / `INTERSECT` / `EXCEPT`, plus multi-row `VALUES`, all share the string-family LCT. - Comparison and `IN`: every participant is cast to that LCT — **including the `IN` left-hand side** (`InTypeCoercion` uses `findWiderCommonType` over `value +: list`). Casting to `CHAR` pads, so unequal `CHAR` lengths compare equal after widen; casting to `VARCHAR`/`STRING` keeps the `CHAR` pad, so `CHAR 'a'` (stored as `'a '`) is not equal to `VARCHAR`/`STRING 'a'` unless the other side carries the same trailing blank. Ignoring trailing blanks is a collation concern (`RTRIM`), not a type-level `PAD SPACE` policy (design D17). ### Why are the changes needed? #58033 left the collated mixed-length LCT gap as a known issue, and had only thin coverage of `UNION ALL` / `INTERSECT` with no compare / `IN`-LHS / `VALUES` / `EXCEPT` matrix. Without pinning the LCT-cast rule, CHAR vs CHAR vs VARCHAR equality is easy to misread as PAD SPACE. ### Does this PR introduce _any_ user-facing change? Yes, when the flag is on: collated `CHAR`/`VARCHAR` values of different lengths but the same collation now widen to `CHAR`/`VARCHAR(max(n,m))` instead of an indeterminate collation. Compare / `IN` / set-op results for mixed `CHAR` lengths follow the LCT-cast rule above (this matches what the engine already did for non-collated paths; the new tests lock it in). ### How was this patch tested? - New `BasicCharVarcharTestSuite` case covering compare, `IN` (including analyzed-plan assertion that the LHS widens), `RTRIM`, collated LCT, `UNION` / `INTERSECT` / `EXCEPT`, and multi-row `VALUES`. - Expanded golden file `charvarchar-standard-semantics.sql`. - Locally: `BasicCharVarcharTestSuite` (20 tests) and the golden file regeneration/run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
