JingsongLi commented on code in PR #1196:
URL: https://github.com/apache/paimon-rust/pull/1196#discussion_r4344980781
##########
crates/paimon/src/io/file_io.rs:
##########
@@ -167,6 +172,28 @@ impl FileFormatMetadataCacheContext {
Err(_) => panic!("file-format metadata cache type mismatch"),
}
}
+
+ /// ROW footer and block index metadata has its own typed cache. Keep it
+ /// separate from the Parquet cache, which also uses `get_or_init`.
+ pub(crate) fn row_metadata_get_or_init<T>(
+ &self,
+ init: impl FnOnce(usize) -> T,
+ ) -> Option<Arc<T>>
+ where
+ T: Send + Sync + 'static,
+ {
+ if self.max_bytes == 0 || self.process_id != std::process::id() {
+ return None;
+ }
+ let cache = Arc::clone(self.row_metadata_cache.get_or_init(|| {
Review Comment:
[P2] Keep ROW and Parquet within the configured metadata-cache budget
This creates an additional cache with min(max_bytes, 32 MiB), while
get_or_init still gives Parquet the full max_bytes. The existing
file-format.metadata-cache.max-size option promises the maximum parsed
file-format metadata cache size. A Data Evolution table can warm both pools
through ordinary Parquet scans and sparse ROW sidecar reads in the same FileIO,
so the setting no longer bounds their combined retention. A local probe with
one FileIO configured to 64 KiB receives 64 KiB for each pool; at the default,
their budgets total 82 MiB against the configured 50 MiB. Please share
accounting or divide the configured budget between formats; an intentionally
additive ROW budget needs its own explicit setting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
