Jackie-Jiang opened a new pull request, #20153: URL: https://github.com/apache/pinot/pull/20153
## Summary `Utf8Utils.encodedLength` delegated to Guava's `Utf8.encodedLength`, which throws `IllegalArgumentException` on an unpaired surrogate, while `Utf8Utils.encode` (`String.getBytes(UTF_8)`) replaces one with a single `?` byte. An unpaired surrogate is reachable from ordinary queries, e.g. `SUBSTR` splitting an emoji, so code that sizes a buffer with `encodedLength` and fills it with `encode` could throw where the encoder would have succeeded. `encodedLengthWithReplacement` already implemented the matching count. This PR collapses the two into a single `encodedLength` whose result is exactly `encode(s).length` for every string: - `encodedLength` counts an unpaired surrogate as the one replacement byte `String.getBytes` writes, and scans the leading ASCII run before entering the general per-char loop. `encodedLengthWithReplacement` is removed; its callers in `ForwardIndexHandler` and `SegmentDictionaryCreator` now call `encodedLength`, and the Guava `Utf8` import is gone from `Utf8Utils`. - `isAscii(byte[])` ORs eight bytes at a time through a `long` view of the array before the byte tail, then checks the high bit of every lane once. - `Utf8UtilsTest` asserts `encodedLength` against both the expected byte count and `encode(s).length` for empty, ASCII, two-byte, three-byte, surrogate-pair, and unpaired high/low surrogate inputs, and covers `isAscii` at every array length and non-ASCII position around the 8-byte stride. `MapUtils.serializedSize` already called `encodedLength`, so its size now agrees with the `encode` call that fills the buffer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
