Rich-T-kid commented on issue #12195: URL: https://github.com/apache/arrow-rs/issues/12195#issuecomment-6827998202
Narrowing this epic to the direct state-transition fix and closing both exploratory sub-issues: - #12196 does not apply to the current scan path. Arrow creates a new record batch reader and `DictionaryBuffer` for each row group, so separately allocated dictionaries from different row groups are never compared in one buffer. - #12197 avoids the spill by returning short batches at encoding boundaries, but that changes batch sizing and can move the cost into downstream coalescing and repartitioning. The remaining direction is to keep the batch shape unchanged and avoid expansion: preserve existing dictionary keys, merge only unique values, and remap incoming keys or intern plain values when the encoding changes within a row group. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
