jiwen624 opened a new pull request, #58235:
URL: https://github.com/apache/spark/pull/58235
### What changes were proposed in this pull request?
`PivotFirst` indexed the pivot values with
`Map(pivotColumnValues.zipWithIndex)`, so values that compare as equal
collapsed onto one entry holding the *last* of their indices while `indexSize`
shrank to the distinct count, leaving the stored index pointing past the end of
the buffer. Every entry now gets its own slot, shared between entries that
compare as equal, and `eval` expands back to one array element per entry.
### Why are the changes needed?
A PIVOT whose `IN` list repeats a value writes outside its allocated slots
in the aggregation buffer:
```sql
SELECT * FROM VALUES (1, 1, 10), (1, 2, 20) AS t(id, k, v)
PIVOT (sum(v) FOR k IN (1 AS x, 1 AS y));
java.lang.AssertionError: index (1) should < 1
```
The bounds check is an assert, so a normal build corrupts the buffer
silently instead of failing. Duplicates are not limited to identical literals:
(0.0D, -0.0D) and ('a', 'A') under UTF8_LCASE reach the same path. The
non-optimized path already answers these queries correctly.
### Does this PR introduce _any_ user-facing change?
Yes. The optimized path now returns one column per listed value, matching
the non-optimized path, instead of failing with AssertionError or silently
corrupting the buffer.
### How was this patch tested?
Added UT cases.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
