Rich-T-kid commented on PR #23187: URL: https://github.com/apache/datafusion/pull/23187#issuecomment-5192617111
@alamb I think the second approach is easier to work with, more reasonable, and less error-prone. But from my understanding, this isn't currently possible, or at least isn't exposed in DataFusion. The only config that currently exists is for writing: https://docs.rs/datafusion/latest/datafusion/config/struct.ParquetOptions.html#structfield.dictionary_enabled > here you tell the parquet reader that the low cardinality columns (that would be good for dictionary encoding) should be read as Dictionary... Arrays I think it'd be nice to expose this as a session prop so users can configure it with SET notation in SQL: ``` SET datafusion.execution.batch_size = 4096; SET datafusion.parquet.dictionary_read_enable; .... ``` The only caveat that comes to mind is this may cause the Parquet reader to read all string columns into dictionary arrays, which isn't what we want for precision. the concept your referring to is actually quite similar to what I raised in this issue: https://github.com/apache/datafusion/issues/22891. Ideally we would be able to determine which columns are low-cardinality and change their type dictionary at physical planning time. with #23187 this should provide a nice perf boost. the only issue would be determining what threshold counts as "low cardinality". cc @asolimando -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
