viirya commented on issue #12045: URL: https://github.com/apache/arrow-rs/issues/12045#issuecomment-6772515447
Closing the loop here: #12046 is closed. Replacing the per-value `extend_from_slice` with fixed-size 16-byte chunk copies is a large win on macOS (about -30% to -50% for 8-48 byte values), where short `memcpy` calls are expensive. On Linux it is roughly neutral in an end-to-end scan, and it is slower than `main` in two cases: - Dictionaries of empty strings: every value went to a zero-length `memcpy` (fixable with an early return). - Dictionaries whose values total fewer than 16 bytes: no value can be copied as a whole chunk without reading past the dictionary, so every value falls back to `memcpy` plus the extra checks. Measured on an M4 Max with a `ParquetRecordBatchReader` scan of 65,536 rows in 1,024-row batches, median of 5 alternating rounds, change vs `main`: | dictionary values | macOS | Linux container | |---|---|---| | all empty | +111% | +168% | | `"Y"` / `"N"` | +7% | +21% | | half empty, half 18-19 B | -13% | +21% | | 8 B | -30% | -4% | | 18-19 B | -31% | +9% | A different approach, such as specializing for dictionaries whose values all have the same length, would avoid both of these cases. It would need to show a gain on Linux, not only on macOS. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
