Rich-T-kid opened a new issue, #11941: URL: https://github.com/apache/arrow-rs/issues/11941
### Is your feature request related to a problem or challenge? Follow-up to #11743 and #11940. `InProgressDictionaryArray` can copy keys directly while inputs share the same values `Arc`. When the values pointer changes, it currently materializes the direct state, buffers the remaining dictionary arrays, and uses `concat` when the output batch is finished. Indexed copies also use `take` until #11940 is implemented. These fallbacks create intermediate arrays and make `concat` merge the value mappings even though the coalescer already sees each mapping and its selected keys. ### Describe the solution you'd like Keep lightweight dictionary partitions instead of invoking `take` or `concat`: - Continue gathering keys and validity while the values `Arc` is unchanged. - When the pointer changes, seal that state as a partition referencing its values array and start a new partition. - At `finish`, build the output values once, create a local-to-output key mapping for each partition, and write the final keys in partition order. - Deduplicate equal values across partitions where appropriate without losing the pairing between each partition's keys and values array. This should preserve row order and dictionary semantics while avoiding intermediate arrays and writing the final output only once. It also needs to handle nulls, repeated values across distinct mappings, key overflow, output reuse, and retained-memory accounting. Benchmark shared mappings and frequent values-`Arc` changes across copy, filter, indexed-copy, and interleave workloads. This is related to #11885, which improves the general dictionary `concat` kernel; this issue is specifically about avoiding that kernel in `BatchCoalescer`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
