alamb commented on PR #23187: URL: https://github.com/apache/datafusion/pull/23187#issuecomment-5190571128
Another thing that might be valuable is to make a variant of the tpch benchmarks, where you tell the parquet reader that the low cardinality columns (that would be good for dictionary encoding) should be read as Dictionary... Arrays -- and then run the queries and see how much improvement this PR gives I actually think that would be helpful in general if your goal is to improve the overall performance of the dictionaries in DataFusion. I will file a ticket with more detail -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
