alamb commented on code in PR #19616:
URL: https://github.com/apache/datafusion/pull/19616#discussion_r2669934002
##########
datafusion/core/src/datasource/listing_table_factory.rs:
##########
@@ -160,7 +161,9 @@ impl TableProviderFactory for ListingTableFactory {
}
None => format!("*.{}", cmd.file_type.to_lowercase()),
};
- table_path = table_path.with_glob(glob.as_ref())?;
+ table_path = table_path
+ .with_glob(glob.as_ref())?
+ .with_table_ref(cmd.name.clone());
Review Comment:
I think we could make table_path maintain with_glob. I did this change in
- https://github.com/jizezhang/datafusion/pull/1
And it seems to have worked well
##########
datafusion-cli/src/functions.rs:
##########
@@ -703,6 +703,23 @@ impl TableFunctionImpl for StatisticsCacheFunc {
}
}
+// Implementation of the `list_files_cache` table function in datafusion-cli.
Review Comment:
❤️
##########
datafusion/execution/src/cache/list_files_cache.rs:
##########
@@ -350,23 +358,40 @@ impl ListFilesCache for DefaultListFilesCache {
state.evict_entries();
}
- fn list_entries(&self) -> HashMap<Path, ListFilesEntry> {
+ fn list_entries(&self) -> HashMap<TableScopedPath, ListFilesEntry> {
let state = self.state.lock().unwrap();
- let mut entries = HashMap::<Path, ListFilesEntry>::new();
+ let mut entries = HashMap::<TableScopedPath, ListFilesEntry>::new();
for (path, entry) in state.lru_queue.list_entries() {
entries.insert(path.clone(), entry.clone());
}
entries
}
+
+ fn drop_table_entries(
+ &self,
+ table_ref: &Option<TableReference>,
+ ) -> datafusion_common::Result<()> {
+ let mut state = self.state.lock().unwrap();
+ let mut table_paths = vec![];
Review Comment:
I am not sure how we would know how many entries we needed to remove here.
I think the reason they need to be copied out is that it isn't possible to
modify the lru_queue while iterating over it as well.
If this becomes a performance problem, I think we could potentially solve it
by actually splitting the cache somehow (eg. a map of maps) so we could drop
all entries for a particular table atomically
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
