jayzhan211 commented on issue #26696: URL: https://github.com/apache/datafusion/issues/26696#issuecomment-6380632988
Dictionaries do only pay off at low cardinality, but nothing enforces that today. `enable_rle_to_dictionary` promotes every string column that has dictionary pages, and the planner can't guarantee low cardinality anyway: Parquet dictionaries are per row group, while a GROUP BY's output holds every distinct value in the partition. So each operator has to remember not to do work proportional to the dictionary size, and some don't, such as `hash_dictionary` and the string functions that map over the dictionary values. Rather than the aggregate-only change proposed here, I'll look for a structural way to handle this, so that no operator has to rely on that assumption. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
