alchemist51 commented on code in PR #18855:
URL: https://github.com/apache/datafusion/pull/18855#discussion_r2548491341
##########
datafusion/execution/src/cache/cache_manager.rs:
##########
@@ -42,8 +47,18 @@ pub type FileStatisticsCache =
/// especially when done over remote object stores.
///
/// See [`crate::runtime_env::RuntimeEnv`] for more details
-pub type ListFilesCache =
- Arc<dyn CacheAccessor<Path, Arc<Vec<ObjectMeta>>, Extra = ObjectMeta>>;
+pub trait ListFilesCache:
Review Comment:
I think one of the things I would like to do in the Cache Manager caches is
to segregate the cache eviction policy. Personally I think the user should be
given an option on what is the eviction behaviour they want. wdyt @alamb
@BlakeOrth ? I can work on getting some draft out this weekend on it.
##########
datafusion/execution/src/cache/cache_manager.rs:
##########
@@ -209,6 +255,8 @@ impl Default for CacheManagerConfig {
Self {
table_files_statistics_cache: Default::default(),
list_files_cache: Default::default(),
+ list_files_cache_limit: DEFAULT_LIST_FILES_CACHE_LIMIT,
+ list_files_cache_ttl: DEFAULT_LIST_FILES_CACHE_TTL,
Review Comment:
Some usecases don't need a TTL, we should provide a way to keep that disable
as well.
##########
datafusion/execution/src/cache/cache_manager.rs:
##########
@@ -228,13 +276,30 @@ impl CacheManagerConfig {
}
/// Set the cache for listing files.
- ///
+ ///
/// Default is `None` (disabled).
- pub fn with_list_files_cache(mut self, cache: Option<ListFilesCache>) ->
Self {
+ pub fn with_list_files_cache(
+ mut self,
+ cache: Option<Arc<dyn ListFilesCache>>,
+ ) -> Self {
self.list_files_cache = cache;
self
}
+ pub fn with_list_files_cache_limit(mut self, limit: usize) -> Self {
+ self.list_files_cache_limit = limit;
+ self
+ }
+
+ pub fn with_list_files_cache_ttl(mut self, ttl: Duration) -> Self {
+ self.list_files_cache_ttl = ttl;
+ if ttl.is_zero() {
Review Comment:
Why are we setting it to None in case of no TTL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
