jiwen624 opened a new pull request, #57272: URL: https://github.com/apache/spark/pull/57272
### What changes were proposed in this pull request? Follow-up to #57177. `approx_top_k` on a non-binary collated string keyed its DataSketches item on getCollationKey(...).toString(). ICU sort keys are arbitrary bytes, not valid UTF-8, so .toString() lossily maps them to U+FFFD and merges collation-distinct values into one item with an inflated count. This changes CollatedString to key on the raw sort-key byte[] (via CollationFactory.getCollationKeyBytes), so key and original round-trip losslessly with no Java String intermediary. The on-wire serde format is unchanged (only original is persisted). ### Why are the changes needed? Wrong counts for any UNICODE/UNICODE_CI/locale collation over non-ASCII data (e.g. 且/丕/世 collapse to one item). UTF8_BINARY and UTF8_LCASE are unaffected. ### Does this PR introduce _any_ user-facing change? Yes — fixes incorrect approx_top_k counts/items for ICU-collated non-ASCII strings. ### How was this patch tested? Added UT case. ### Was this patch authored or co-authored using generative AI tooling? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
