Rich-T-kid commented on issue #24117: URL: https://github.com/apache/datafusion/issues/24117#issuecomment-5673161387
Taking a glance, and I agree with your analysis for the most part, @yinli-systems. I think #25185 should be a pretty big factor in the performance boost we expect to gain from dictionary encoding. The narrowed goal of this epic is to support CBDE for parquet files, which gives us the privilege of working with the arrow-rs parquet reader. From my understanding, one row group maps to a dictionary values array, in aggregations, we can move work away from `O(row_count)` to `O(row_group_set)`, which would be huge. with the linked PR we would run computation once and then cache the results even if we recieve thousands of .`intern()` calls. I took a brief look some time ago, but it may be worth looking at other operators/kernels to see if there are any dictionary-specific optimizations that can be applied to give us more leeway to decide between `utf8` and `dict<_,utf8>`. once #24227 is shipped it will also be easier to benchmark what changes are moving the needle in terms of performance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
