alamb commented on code in PR #26751: URL: https://github.com/apache/datafusion/pull/26751#discussion_r4391720600
########## dev/update_config_docs.sh: ########## @@ -190,14 +192,36 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; ``` [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +For point lookups, the following techniques also help: + +- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered). + This makes techniques such as Parquet min/max metadata pruning more effective. +- Enable `datafusion.execution.parquet.pushdown_filters` so filters are + evaluated while decoding Parquet. +- Set the Parquet metadata cache size via + `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the + files you query repeatedly. +- Reuse prepared statements to avoid re-planning each query. Review Comment: done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
