Tushar7012 commented on code in PR #19975:
URL: https://github.com/apache/datafusion/pull/19975#discussion_r2728888632
##########
datafusion/catalog-listing/src/table.rs:
##########
@@ -712,20 +712,21 @@ impl ListingTable {
});
};
// list files (with partitions)
- let file_list =
future::try_join_all(self.table_paths.iter().map(|table_path| {
- pruned_partition_list(
- ctx,
- store.as_ref(),
- table_path,
- filters,
- &self.options.file_extension,
- &self.options.table_partition_cols,
- )
- }))
- .await?;
let meta_fetch_concurrency =
ctx.config_options().execution.meta_fetch_concurrency;
- let file_list =
stream::iter(file_list).flatten_unordered(meta_fetch_concurrency);
+ let file_list = stream::iter(self.table_paths.iter())
Review Comment:
Yes, you're right - this was committed by accident. I've reverted the
[table.rs](vscode-file://vscode-app/c:/Users/td334/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html)
changes in commit 9302c3b1a. These changes belong in a separate PR for
parallelizing list_files_for_scan. Thanks for catching it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
