Rich-T-kid commented on PR #24227: URL: https://github.com/apache/datafusion/pull/24227#issuecomment-5698213418
> These results reproduce the earlier performance direction. I think the implementation needs to avoid automatically promoting high-cardinality or dictionary-fallback columns. @kumarUjjawal That is expected with this implementation. The larger epic #24111, specifically #24117, will handle deciding on a per-query basis whether to read the column in as Dictionary or as Utf8/Utf8View. The point of this feature flag is to let users enable direct RLE_DICTIONARY reads from Parquet files in DataFusion, for cases where they know the shape of their data and query well enough to make that decision themselves. By default, this flag will be turned off. This line of thinking is in line with what alamb mentioned here https://github.com/apache/datafusion/issues/24111#issuecomment-5206277128 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
