alamb commented on code in PR #26751: URL: https://github.com/apache/datafusion/pull/26751#discussion_r4391715701
########## dev/update_config_docs.sh: ########## @@ -190,14 +192,36 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; ``` [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +For point lookups, the following techniques also help: Review Comment: it is a good idea -- I pushed some changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
