BlakeOrth commented on code in PR #18855:
URL: https://github.com/apache/datafusion/pull/18855#discussion_r2550554230
##########
datafusion/execution/src/cache/cache_manager.rs:
##########
@@ -209,6 +255,8 @@ impl Default for CacheManagerConfig {
Self {
table_files_statistics_cache: Default::default(),
list_files_cache: Default::default(),
+ list_files_cache_limit: DEFAULT_LIST_FILES_CACHE_LIMIT,
+ list_files_cache_ttl: DEFAULT_LIST_FILES_CACHE_TTL,
Review Comment:
I actually agree with your first assessment that TTL can, and likely should,
be optional if a user has a use case where they know the underlying objects are
immutable once written (my personal use of DataFusion falls into this
category). In either case, we may have to accept some differences between this
cache and the Metadata cache. Unlike the Metadata cache, which can issue HEAD
requests against objects to detect modification, there's no mechanism in an
object store to inquire whether or not paths/sub-paths have been changed, so
this cache will need to make some concessions around that limitation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
