yadavay-amzn opened a new pull request, #56215:
URL: https://github.com/apache/spark/pull/56215
### What changes were proposed in this pull request?
Make `PivotFirst` use collation-aware key comparison when the pivot column
has a non-binary collation (e.g., `UTF8_LCASE`, `UNICODE_CI`).
The fix transforms pivot value keys via `CollationFactory.getCollationKey()`
before inserting into / looking up from the `pivotIndex` HashMap, so that
values equal under the column's collation map to the same index.
### Why are the changes needed?
`PivotFirst` uses a `HashMap` with default (binary) equality for pivot value
lookup. When the pivot column has a case-insensitive or accent-insensitive
collation, values that should match (e.g., `'SALES'` and `'sales'` under
`UTF8_LCASE`) are treated as different keys, producing incorrect NULLs in the
pivoted output.
Repro:
```sql
CREATE TABLE t (dept STRING COLLATE UTF8_LCASE, amount INT) USING PARQUET;
INSERT INTO t VALUES ('SALES', 100), ('sales', 50);
SELECT * FROM t PIVOT (SUM(amount) FOR dept IN ('sales' AS sales));
-- Expected: 150
-- Actual (before fix): NULL
```
### Does this PR introduce _any_ user-facing change?
Yes — PIVOT queries on columns with non-binary collations now correctly
match values according to the column's collation rules instead of producing
incorrect NULLs.
### How was this patch tested?
Added 7 new tests to `DataFramePivotSuite`:
- UTF8_LCASE collation (case-insensitive matching)
- UNICODE_CI collation
- Multiple pivot values with mixed case
- Binary collation preserved (no regression — different cases remain
distinct)
- Non-string types (INT) still work
- NULL handling in pivot column
- Empty string pivot values
All 40 existing tests in `DataFramePivotSuite` continue to pass.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
