ethan-tyler commented on PR #9220:
URL: https://github.com/apache/arrow-rs/pull/9220#issuecomment-3797175391
> Why wouldn't we just pack to Utf8View directly?
Good question @alamb - Packing directly to Utf8View would require a
dictionary builder for view types (dedup and incremental construction over a
block/buffer-indexed view layout, plus view invariants like prefix/offset
correctness).
The two step path reuses the existing Dictionary(K, Utf8/Binary) packing,
then reuses the existing cast machinery to produce Dictionary(K,
Utf8View/BinaryView). When the dictionary values are Utf8/Binary and offsets
fit, the cast can build views over the existing values buffer via
append_block/view_from_dict_values (no extra value buffer copy).
For Large* / oversized values, it can fall back to the general cast path
(potentially copying into view blocks) instead of trying to force a zero copy
view representation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
