github-actions[bot] commented on code in PR #68238:
URL: https://github.com/apache/doris/pull/68238#discussion_r4056508408
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonMetadataOps.java:
##########
@@ -436,14 +436,10 @@ public boolean tableExist(String dbName, String tblName) {
@Override
public boolean databaseExist(String dbName) {
try {
- return executionAuthenticator.execute(() -> {
- try {
- catalog.getDatabase(dbName);
- return true;
- } catch (DatabaseNotExistException e) {
- return false;
- }
- });
+ // CachingCatalog has no public database-cache invalidation API.
listDatabases is
+ // delegated to the underlying catalog, so existence checks after
REFRESH do not
+ // reuse a Database object cached before an external drop or
recreate.
+ return executionAuthenticator.execute(() ->
catalog.listDatabases().contains(dbName));
Review Comment:
[P2] Preserve Paimon's point-lookup semantics in this cache bypass. In
1.4.2, `getDatabase("sys")` resolves the virtual system database, and Hive
catalogs can resolve names case-insensitively; backing `listDatabases()`
results neither add `sys` nor guarantee the caller's spelling. Exact membership
therefore returns false for names the catalog considers present, skips
`performCreateDb`'s intended early branch, and can make `CREATE DATABASE IF NOT
EXISTS sys` throw or a differently cased existing Hive database reach property
validation. Handle the system database explicitly and compare listed names
according to `catalog.caseSensitive()`, with regressions for both cases.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalCatalog.java:
##########
@@ -130,6 +132,52 @@ public Table getPaimonTable(NameMapping nameMapping,
String branch, String query
return loadPaimonTable(nameMapping, queryType, identifier);
}
+ public synchronized void invalidatePaimonTable(NameMapping nameMapping)
throws Exception {
+ // Property changes reset and close the SDK catalog before retiring
Doris cache entries.
+ // Do not recreate that catalog merely to invalidate an already
retired generation.
+ if (!isInitialized()) {
+ return;
+ }
+ Identifier identifier = tableIdentifier(nameMapping, null, null);
+ executionAuthenticator.execute(() -> {
+ catalog.invalidateTable(identifier);
+ return null;
+ });
+ }
+
+ public synchronized void invalidatePaimonDatabase(String remoteDbName)
throws Exception {
+ invalidateCachedPaimonTables(identifier ->
identifier.getDatabaseName().equals(remoteDbName));
+ }
+
+ public synchronized void invalidatePaimonCatalog() throws Exception {
+ invalidateCachedPaimonTables(ignored -> true);
+ }
+
+ private void invalidateCachedPaimonTables(Predicate<Identifier> predicate)
throws Exception {
+ // A property ALTER closes the old SDK catalog before Doris retires
its cache entries.
+ // The new SDK catalog must remain lazily initialized in that callback.
+ if (!isInitialized() || !(catalog instanceof CachingCatalog)) {
+ return;
+ }
+ CachingCatalog cachingCatalog = (CachingCatalog) catalog;
+ List<Identifier> cachedIdentifiers = new
ArrayList<>(cachingCatalog.tableCache().asMap().keySet());
Review Comment:
[P1] Fence SDK loads that overlap refresh. These invalidators synchronize on
`PaimonExternalCatalog`, but `getPaimonTable`/`loadPaimonTable` do not take
that monitor. Paimon 1.4.2's `CachingCatalog.getTable` checks the cache,
performs the delegated load, and only then publishes with `tableCache.put`; an
SDK-only TVF or table-existence miss that starts before REFRESH can therefore
be absent from this key snapshot (or the one-shot table invalidation) and
insert its pre-refresh handle after invalidation returns. With no Doris
`tableEntry` to force a reload, later direct requests keep reusing that handle,
preserving the stale-after-refresh failure. Serialize SDK publication against
all three invalidators or fence loads with an invalidation generation, and add
a paused-miss regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
