github-actions[bot] commented on code in PR #68238:
URL: https://github.com/apache/doris/pull/68238#discussion_r4056151347
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java:
##########
@@ -118,6 +121,38 @@ public Table getPaimonTable(NameMapping nameMapping) {
return
tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getPaimonTable();
}
+ @Override
+ public void invalidateTable(long catalogId, String dbName, String
tableName) {
+ try {
+ invalidatePaimonTables(catalogId, nameMapping ->
matchTable(nameMapping, dbName, tableName));
+ } finally {
+ super.invalidateTable(catalogId, dbName, tableName);
+ }
+ }
+
+ @Override
+ public void invalidateDb(long catalogId, String dbName) {
+ try {
+ invalidatePaimonTables(catalogId, nameMapping ->
matchDb(nameMapping, dbName));
+ } finally {
+ super.invalidateDb(catalogId, dbName);
+ }
+ }
+
+ private void invalidatePaimonTables(long catalogId, Predicate<NameMapping>
predicate) {
+ MetaCacheEntry<NameMapping, PaimonTableCacheValue> tables =
tableEntry.getIfInitialized(catalogId);
+ if (tables == null) {
+ return;
+ }
+ List<NameMapping> nameMappings = new ArrayList<>();
+ tables.forEach((nameMapping, ignored) -> {
+ if (predicate.test(nameMapping)) {
+ nameMappings.add(nameMapping);
+ }
+ });
+ nameMappings.forEach(tableLoader::invalidate);
+ }
Review Comment:
[P1] Invalidate direct Paimon SDK handles that never enter tableEntry.
PaimonTableValuedFunction calls PaimonExternalCatalog.getPaimonTable directly,
which uses catalog.getTable and bypasses this Doris cache. In a TVF-only
workload the Paimon engine group/tableEntry is absent, so
ExternalMetaCacheMgr.safeInvalidate skips this path (or this helper sees no
mappings) and the CachingCatalog handle survives REFRESH
TABLE/DATABASE/CATALOG. The next TVF can therefore still observe an externally
dropped or changed table. Route direct loads through the managed loader or add
equivalent SDK invalidation, with a TVF-only refresh regression.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java:
##########
@@ -118,6 +121,38 @@ public Table getPaimonTable(NameMapping nameMapping) {
return
tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getPaimonTable();
}
+ @Override
+ public void invalidateTable(long catalogId, String dbName, String
tableName) {
+ try {
+ invalidatePaimonTables(catalogId, nameMapping ->
matchTable(nameMapping, dbName, tableName));
+ } finally {
+ super.invalidateTable(catalogId, dbName, tableName);
+ }
+ }
+
+ @Override
+ public void invalidateDb(long catalogId, String dbName) {
+ try {
+ invalidatePaimonTables(catalogId, nameMapping ->
matchDb(nameMapping, dbName));
+ } finally {
Review Comment:
[P1] Invalidate Paimon's database cache for database refresh. This path only
calls catalog.invalidateTable for table mappings, but Paimon CachingCatalog
keeps Database objects in a separate databaseCache that invalidateTable never
evicts. PaimonMetadataOps.databaseExist (used by CREATE DATABASE) calls
catalog.getDatabase and can therefore return the pre-refresh object after an
external database drop/recreate, even when table entries were enumerated
successfully. Add database-cache invalidation/rebuild coverage for REFRESH
DATABASE/CATALOG and a regression for getDatabase/CREATE after an out-of-band
change.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -130,6 +130,15 @@ public Table getPaimonTable(NameMapping nameMapping,
String branch, String query
return loadPaimonTable(nameMapping, queryType, identifier);
}
+ public void invalidatePaimonTable(NameMapping nameMapping) throws
Exception {
+ makeSureInitialized();
+ Identifier identifier = tableIdentifier(nameMapping, null, null);
+ executionAuthenticator.execute(() -> {
Review Comment:
[P1] Do not initialize the SDK from this invalidation callback.
CatalogMgr.modifyCatalogProps resets and closes the old Paimon catalog, then
onCatalogOperationalContextChanged reaches this method while the Doris cache
group still contains old mappings. makeSureInitialized therefore eagerly
constructs/authenticates the newly configured catalog solely to invalidate
entries in the already-closed instance; a failed initialization can make the
committed ALTER/replay fail after the old context is gone, and a successful one
defeats the intended lazy next-statement rebuild. Skip SDK invalidation while
uninitialized (or retire the group first) and let the next load create the
fresh generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
