JingsongLi commented on code in PR #4565:
URL: https://github.com/apache/paimon/pull/4565#discussion_r1853475831
##########
paimon-core/src/main/java/org/apache/paimon/Snapshot.java:
##########
@@ -437,4 +413,41 @@ public enum CommitKind {
/** Collect statistics. */
ANALYZE
}
+
+ // =================== Utils for reading =========================
+
+ public static Snapshot fromJson(String json) {
+ return JsonSerdeUtil.fromJson(json, Snapshot.class);
+ }
+
+ public static Snapshot fromPath(FileIO fileIO, Path path) {
+ try {
+ return tryFromPath(fileIO, path);
+ } catch (FileNotFoundException e) {
+ String errorMessage =
+ String.format(
+ "Snapshot file %s does not exist. "
+ + "It might have been expired by other
jobs operating on this table. "
+ + "In this case, you can avoid concurrent
modification issues by configuring "
+ + "write-only = true and use a dedicated
compaction job, or configuring "
+ + "different expiration thresholds for
different jobs.",
+ path);
+ throw new RuntimeException(errorMessage, e);
+ }
+ }
+
+ public static Snapshot tryFromPath(FileIO fileIO, Path path) throws
FileNotFoundException {
+ try {
+ Snapshot snapshot = SNAPSHOT_CACHE.getIfPresent(path);
+ if (snapshot == null) {
+ snapshot = Snapshot.fromJson(fileIO.readFileUtf8(path));
+ SNAPSHOT_CACHE.put(path, snapshot);
+ }
+ return snapshot;
+ } catch (FileNotFoundException e) {
Review Comment:
`tryFromPath` is just for `FileNotFoundException`, invoker can know the file
does not exists.
##########
paimon-core/src/main/java/org/apache/paimon/catalog/CachingCatalog.java:
##########
@@ -316,4 +319,36 @@ public void refreshPartitions(Identifier identifier)
throws TableNotExistExcepti
partitionCache.put(identifier, result);
}
}
+
+ // ====================== cache for snapshot and schema files
================================
+
+ public static final Cache<Path, Snapshot> SNAPSHOT_CACHE =
+ Caffeine.newBuilder()
+ .softValues()
+ .expireAfterAccess(Duration.ofMinutes(10))
+ .maximumSize(300)
+ .executor(Runnable::run)
+ .build();
+
+ public static final Cache<Path, TableSchema> SCHEMA_CACHE =
+ Caffeine.newBuilder()
+ .softValues()
+ .expireAfterAccess(Duration.ofMinutes(10))
+ .maximumSize(100)
+ .executor(Runnable::run)
+ .build();
+
+ public static void invalidateMetaCacheForPrefix(Path tablePath) {
+ String path = tablePath.toString();
+ invalidateMetaCacheForPrefix(SNAPSHOT_CACHE, path);
+ invalidateMetaCacheForPrefix(SCHEMA_CACHE, path);
+ }
+
+ private static void invalidateMetaCacheForPrefix(Cache<Path, ?> cache,
String tablePath) {
Review Comment:
deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@paimon.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
