rizaon commented on code in PR #4518:
URL: https://github.com/apache/iceberg/pull/4518#discussion_r962269646
##########
core/src/main/java/org/apache/iceberg/ManifestFiles.java:
##########
@@ -43,6 +47,30 @@ private ManifestFiles() {}
GenericManifestFile.class.getName(),
ManifestFile.PARTITION_SUMMARY_TYPE,
GenericPartitionFieldSummary.class.getName()));
+ private static final int CONTENT_CACHES_EXPIRATION_INTERVAL_SECOND = 60 *
60; // 1 hour
Review Comment:
[f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af)
change `CONTENT_CACHE` to evict by size and reference.
Since content caching is per catalog/FileIO, different FileIO might have
different caching configuration, including expiration time. This expiration
time can be longer than 60s. Thus, using time-based eviction for
`CONTENT_CACHE` might conflict with with per-FileIO cache configuration. I
don't think we can use maximum weight as well for `CONTENT_CACHE` since
according to the
[documentation](https://github.com/ben-manes/caffeine/wiki/Eviction) entry's
weight is only calculated on entry creation and update time, and static
thereafter. All `CONTENT_CACHE` entries then will have 0 weight initially and
potentially stay that way.
[f76fa5d](https://github.com/apache/iceberg/pull/4518/commits/f76fa5d7f8257c63fd90df32fd841ac989cb26af)
add default upper bound to 8 FileIO to cache (no specific reason about why it
should be 8, I'm open to change it). Additionally, it also use `weakKeys()` to
allow garbage collection of an entry if there are no other strong references to
the keys.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
