Rich-T-kid opened a new issue, #12205: URL: https://github.com/apache/arrow-rs/issues/12205
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The IPC stream writer serializes the entire values array of a dictionary column, even when the keys in the record batch only reference a small prefix. This can copy and encode a large amount of unreachable data after filtering or slicing a dictionary array, and the cost is amplified when IPC is used for spill files. **Describe the solution you would like** For IPC streams, inspect the dictionary keys and encode only the values prefix through the largest referenced key. Preserve the existing behavior for IPC files, which cannot later extend or replace a dictionary safely. Add a focused writer benchmark where a small batch references only the beginning of a large dictionary so the current overhead and any improvement are measurable. **Describe alternatives you have considered** Callers can compact dictionaries before writing, but that duplicates work at every call site and is easy to miss. Fully remapping sparse keys could reduce the dictionary further, but prefix trimming is a smaller change that does not rewrite keys. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
