Rich-T-kid opened a new pull request, #12208: URL: https://github.com/apache/arrow-rs/pull/12208
# Which issue does this PR close? - Closes #12205 # Rationale for this change IPC streams currently write every dictionary value even when the record-batch keys reference only a small prefix. This can write a large amount of unreachable data after filtering or slicing a dictionary array. # What changes are included in this PR? For stream writers, inspect the dictionary keys and encode only the values through the largest referenced key. Later batches can extend that prefix through the existing delta or resend handling. File writers retain the existing full-dictionary behavior because files cannot safely extend or replace a dictionary after it has been written. # Are these changes tested? A regression test covers growing, shrinking, repeated, and null-only key sets under both delta and resend handling, and retains stream/file round-trip parity. The focused benchmark added in #12207 uses 8,192 keys referencing 16 entries from 65,536 256-byte values: | | Time | |---|---:| | Current main | 219.53 us | | This PR | 5.93 us | This is a 97.3% reduction for the benchmarked stream write. A DataFusion ClickBench Q33 forced-spill run with RLE-to-dictionary enabled did not show a stable end-to-end change above run variance. # Are there any significant user-facing changes? No public API change. IPC streams may be smaller while decoding to the same logical arrays. # AI usage disclosure Codex helped implement the change, add the regression coverage, and run the benchmark and DataFusion spill validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
