mbutrovich opened a new pull request, #12289: URL: https://github.com/apache/arrow-rs/pull/12289
# Which issue does this PR close? N/A. # Rationale for this change Engines built on arrow-rs encode string keys with `RowConverter` on every batch in several places. DataFusion does it when it merges sorted runs on a multi-column key, in TopK, and in range repartitioning. DataFusion Comet does it for range partitioning in its shuffle writer. A profile of `convert_columns` on strings with lengths that vary from row to row spent about 30% of its samples in `memmove`, called once per value to copy its last, partial block. `Utf8View` and `BinaryView` also encode through the generic `ArrayIter` path rather than a loop over their views. # What changes are included in this PR? - The partial last block of each variable-length value is written with a few fixed-size copies instead of `copy_from_slice`, whose length the compiler can't see. This removes the `memcpy` call from `variable::encode_one` on aarch64 and x86-64, which also reduces the callee-saved registers its prologue saves on every call. Strings, binary, string dictionaries, and lists all encode their values through `encode_one`. - `Utf8View` and `BinaryView` encode with a loop over the views that reads a value of 12 bytes or fewer from its view. - New `row_format` cases with random-length `Utf8` at 4096 rows, and random-length `Utf8` and `Utf8View` at 8192 rows, DataFusion's default batch size. One `Utf8View` case has only values that fit in their views. ## Benchmarks TODO: benchmark bot results (Neoverse-V2 and Xeon, default and aligned builds). ## Trade-offs - The partial-block copy branches on the remaining length. With random lengths those branches mispredict, but they cost less than the `memcpy` call they replace. - Values whose length is a multiple of 8 bytes, such as `string(16)`, never copy a partial block, so they gain little. # Are these changes tested? Yes. `test_variable_width_every_length` in `arrow-row/src/lib.rs` encodes every length from 0 to 101 bytes for `Binary` and `BinaryView`, ascending and descending, with distinct bytes. It checks that rows decode to the input and order like the values. The existing fuzz tests cover views with nulls and with several data buffers. # Are there any significant user-facing changes? No. The encoded rows are byte-for-byte the same. # AI usage disclosure I wrote this PR with help from Claude but understand and support the core approach, implementation, and test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
