Rich-T-kid commented on issue #12196: URL: https://github.com/apache/arrow-rs/issues/12196#issuecomment-6827915257
I prototyped the equality check and tested it through DataFusion with RLE-to-dictionary reads enabled. The test file had 4,194,304 rows split across 1,024 independently dictionary-encoded row groups, including a case where every row group used the same dictionary ordering. There was no measurable change: 2.048s before and 2.050s after for the identical-dictionary case. The reason is that the premise here does not match the current reader lifetime. DataFusion may assign many row groups to one scan partition, but arrow-rs constructs a separate `ParquetRecordBatchReader` (and therefore a new `DictionaryBuffer`) for each row group. The old buffer is dropped before the next row group and its separately allocated dictionary are read, so `as_keys_with_bit_width` never compares those two dictionaries. Mixed dictionary/plain data pages within one row group can still make a live `DictionaryBuffer` spill. The batch-boundary experiment in #12197 and the broader state-transition work in #12195 still apply to that case. Closing this issue because comparing equal dictionaries across allocations does not currently affect the DataFusion Parquet scan path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
