adriangb opened a new issue, #12183: URL: https://github.com/apache/arrow-rs/issues/12183
**Describe the bug** The IPC writer writes every child of a sliced dense `UnionArray` in full. A slice of 10 rows from a large dense union costs about as much as the whole parent. The writer already narrows other sliced types. `List`, `LargeList` and `Map` rebase their offsets and slice the child. Sparse `Union` slices each child to the parent slice. `RunEndEncoded` unslices the run array. Dense `Union` slices only the type ids and offsets buffers. **To Reproduce** Build a dense union of 100,000 rows that alternate between a nullable `Int32` child and a nullable `Utf8` child. Write it with `StreamWriter`, then write `slice(50_001, 10)`. | Array | Whole stream (bytes) | 10-row slice (bytes) | | --- | --- | --- | | Dense union of Int32 and Utf8, 100,000 rows | 1,105,624 | 605,680 | | Dense union of Int32 and List<Utf8>, 20,000 rows | 280,440 | 180,496 | The slice drops only the type ids and offsets. All child data is still written. **Expected behavior** The writer narrows each child to the rows that the slice references and rebases the offsets. A child that the slice does not reference is written empty. The data reads back logically equal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
