mightsleep opened a new pull request, #26599: URL: https://github.com/apache/datafusion/pull/26599
## Which issue does this PR close? N/A. Follow-up to #26413, which added the non-ASCII benchmark cases. ## Rationale for this change For non-ASCII strings `left` and `right` find the character boundary with `char_indices().nth()` / `nth_back()`, one codepoint at a time. Counting codepoints is counting the bytes that are not `0b10xx_xxxx`, which can be done a word or a block at a time. ## What changes are included in this PR? - `byte_offset_of_char` counts codepoint starts in 64-byte blocks (a `u8` fold, which LLVM vectorizes on baseline x86-64), then in 8-byte words with byte prefix counts from one multiply, and selects the byte inside the word without a loop. Below n = 16 it skips the blocks. - `byte_offset_of_char_from_end` does the same from the end, replacing `nth_back` in `right` and negative `n`. - The ASCII fast paths are unchanged; the scans are `#[inline(never)]` so they do not slow them down. ## What is the testing strategy for this PR? New unit tests compare both scans with `char_indices` on strings mixing 1 to 4 byte codepoints, with lengths crossing the 8 and 64 byte boundaries, and check extreme `n`. Existing tests pass. `unicode_expressions` bench, main / this PR, min of three interleaved rounds, default target, Zen 5: | case | Utf8 | Utf8View | |---|---|---| | non_ascii long_result | 7.7x | 13.9x | | non_ascii negative_n | 7.4 to 7.6x | 12.6 to 13.9x | | non_ascii short_result | 1.29x | 1.17 to 1.34x | | ASCII cases | 0.98 to 1.02x | 1.07 to 1.16x | ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
