adriangb opened a new pull request, #12186: URL: https://github.com/apache/arrow-rs/pull/12186
# Which issue does this PR close? - Closes #12183. # Rationale for this change The IPC writer writes every child of a sliced dense `Union` in full. A 10-row slice of a large dense union costs about as much as the whole parent. `List`, `LargeList`, `Map` and sparse `Union` are already narrowed to the slice. # What changes are included in this PR? A new `get_dense_union_buffers` narrows each child to the range of rows that the slice references and rebases the offsets to the start of that range. A child that the slice does not reference is written empty. Rows inside a range that no slot references are still written. This keeps every child a zero-copy slice. Stream sizes for a 10-row slice, with 8-byte alignment: | Array | Whole | Slice before | Slice after | | --- | --- | --- | --- | | Dense union of Int32 and Utf8, 100,000 rows | 1,105,624 | 605,680 | 648 | | Dense union of Int32 and List<Utf8>, 20,000 rows | 280,440 | 180,496 | 784 | # Are these changes tested? Yes. New tests write a whole union and a slice, check that both read back equal, and check that the slice is much smaller. They cover nullable children, non-contiguous type ids, a dense union that contains a list of strings, and a child that the slice does not reference. One test also asserts the exact rebased offsets and child ranges. # Are there any significant user-facing changes? No. Sliced dense unions produce smaller IPC output. # AI usage disclosure Claude Code wrote the fix and the tests. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
