Rich-T-kid opened a new issue, #11118: URL: https://github.com/apache/arrow-rs/issues/11118
**Is your feature request related to a problem or challenge?** Part of https://github.com/apache/arrow-rs/issues/7761 `SparseUnion` goes through `GenericInProgressArray` → `concat_fallback` → `MutableArrayData`. The MutableArrayData `build_extend_sparse` copies type_ids in batch but extends ALL child arrays for EVERY row (since SparseUnion has all children at full length). A specialized implementation can avoid MutableArrayData dispatch. **Describe the solution you'd like** ```rust pub(crate) struct InProgressSparseUnionArray { union_fields: UnionFields, type_ids: Vec<i8>, children: Vec<Box<dyn InProgressArray>>, } ``` **`copy_rows(offset, len)` logic:** 1. Append type_ids[offset..offset+len] in one extend_from_slice call 2. For EACH child: call `child.copy_rows(offset, len)` — all children must have the same length in a SparseUnion **Why better than MutableArrayData:** - type_id copy: `extend_from_slice` vs. per-row boxed-closure dispatch - Children: each child uses its specialized InProgressArray instead of MutableArrayData - Pre-allocated buffers vs. dynamic Vec growth in MutableArrayData **Implementation note:** Sparse union semantically requires ALL children to store a value for every logical row (most are null/zero), so there's no way to avoid the O(n_rows × n_children) data cost. The gain is purely in removing MutableArrayData overhead. **Note:** Sparse union is uncommon in practice. Prioritize after all other types. **Benchmarks** ``` cargo bench --bench coalesce_kernels --features test_utils -- sparse_union ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
