This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 0e0ef7f468c branch-4.1: [fix](hive) invalidate stale file cache on
partition refresh when partition cache misses #65334 (#66259)
0e0ef7f468c is described below
commit 0e0ef7f468c16a9842a34dd7d0ce548c2155554a
Author: 924060929 <[email protected]>
AuthorDate: Thu Jul 30 20:51:09 2026 +0800
branch-4.1: [fix](hive) invalidate stale file cache on partition refresh
when partition cache misses #65334 (#66259)
### What problem does this PR solve?
Cherry-pick from master #65334.
Problem Summary:
Partition-level refresh / invalidation
(`HiveExternalMetaCache.invalidatePartitionCache`) returned early when
the target partition was absent from the partition metadata cache —
evicted under `max_hive_partition_cache_num` pressure, or simply never
loaded — and left that partition's file listing cache untouched.
The partition cache (`partition`) and the file listing cache (`file`)
are independent caches with independent size limits
(`max_hive_partition_cache_num` vs `max_external_file_cache_num`) and
evict separately, so the partition entry can be evicted while its file
listing survives. In that state a partition-level refresh cannot rebuild
the exact `FileCacheKey` (it needs the partition path / input format
that only the cached `HivePartition` carries), so the old code gave up
and the stale file listing remained. A later query would reload the
partition metadata and re-hit the same stale file listing — i.e. "the
partition exists, but Doris keeps seeing the old files/data".
Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a
predicate-based invalidation on the file cache keyed by `table id +
partition values`, so the stale listing for that partition is cleared
without over-invalidating other partitions. This mirrors table-level
invalidation, which already invalidates the file cache by predicate.
The partition invalidation coordinator is also decoupled from
`ExternalTable` to `NameMapping` (mirroring the existing
`invalidateTableCache(NameMapping)`) so the path can be unit tested
directly.
This behavior has existed since partition-level invalidation was first
introduced in #15702 (the file cache was only invalidated inside the `if
(partition != null)` branch), and was later carried into the current
`HiveExternalMetaCache` by the #60937 cache-framework refactor.
#### Backport notes
Conflicts resolved by hand, because branch-4.1 does not have #64867
("Key Hive partition cache by name"):
- `HivePartitionValues` on this branch is still keyed by local partition
id (`partitionNameToIdMap` / `partitionValuesMap`) instead of master's
`nameToPartitionValues`, so the removed block looks different. The
replacement is unchanged: `HiveUtil.toPartitionValues(partitionName)`.
This is behavior-preserving on the happy path — on this branch
`partitionValuesMap` is built from
`PartitionKey.createListPartitionKeyWithTypes(..., isHive=true)`, whose
`getPartitionValuesAsStringList()` returns `originHiveKeys`, i.e.
exactly the strings `HiveUtil.toPartitionValues(partitionName)` produced
in the first place.
- The two master-only unit tests that git pulled into the conflict hunk
as context (`testDefaultSpecsFollowConfig`,
`testHivePartitionValuesCopyKeepsIndependentNameMaps`) are not part of
#65334 and were dropped; only the new test from #65334 is added.
### Release note
Fix a Hive external catalog consistency issue where a partition-level
refresh could leave a stale file listing cached when the partition
metadata cache had been evicted, causing queries to keep seeing outdated
files for that partition.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test
- [ ] No need to test
- Behavior changed:
- [ ] No.
- [x] Yes. Partition-level refresh now also invalidates the file listing
cache when the partition metadata cache entry is missing (previously it
was left stale).
- Does this need documentation?
- [x] No.
---
.../datasource/hive/HiveExternalMetaCache.java | 46 ++++++++--------
.../datasource/hive/HiveMetaStoreCacheTest.java | 64 ++++++++++++++++++++++
2 files changed, 87 insertions(+), 23 deletions(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
index 2ba8fe2459b..b52ef1f92a2 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java
@@ -582,7 +582,11 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
}
public void invalidatePartitionCache(ExternalTable dorisTable, String
partitionName) {
- partitionCacheCoordinator.invalidatePartitionCache(dorisTable,
partitionName);
+ invalidatePartitionCache(dorisTable.getOrBuildNameMapping(),
partitionName);
+ }
+
+ public void invalidatePartitionCache(NameMapping nameMapping, String
partitionName) {
+ partitionCacheCoordinator.invalidatePartitionCache(nameMapping,
partitionName);
}
/**
@@ -614,38 +618,34 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
}
private final class PartitionCacheCoordinator {
- private void invalidatePartitionCache(ExternalTable dorisTable, String
partitionName) {
- NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
+ private void invalidatePartitionCache(NameMapping nameMapping, String
partitionName) {
long catalogId = nameMapping.getCtlId();
- MetaCacheEntry<PartitionValueCacheKey, HivePartitionValues>
partitionValuesEntry =
- partitionValuesEntryIfInitialized(catalogId);
MetaCacheEntry<PartitionCacheKey, HivePartition> partitionEntry =
partitionEntryIfInitialized(catalogId);
MetaCacheEntry<FileCacheKey, FileCacheValue> fileEntry =
fileEntryIfInitialized(catalogId);
- if (partitionValuesEntry == null || partitionEntry == null ||
fileEntry == null) {
+ if (partitionEntry == null || fileEntry == null) {
return;
}
long tableId = Util.genIdByName(nameMapping.getLocalDbName(),
nameMapping.getLocalTblName());
- PartitionValueCacheKey key = new
PartitionValueCacheKey(nameMapping, null);
- HivePartitionValues partitionValues =
partitionValuesEntry.getIfPresent(key);
- if (partitionValues == null) {
- return;
- }
-
- Long partitionId =
partitionValues.partitionNameToIdMap.get(partitionName);
- if (partitionId == null) {
- return;
- }
-
- List<String> values =
partitionValues.partitionValuesMap.get(partitionId);
- if (values == null) {
- return;
- }
+ // Derive the partition values directly from the partition name
(the partition-values cache is
+ // populated the same way, via HiveUtil.toPartitionValues) instead
of reading them from that
+ // cache, so file cache invalidation still runs when the table's
partition-values cache entry has
+ // been evicted while its file listings are still cached.
+ List<String> values = HiveUtil.toPartitionValues(partitionName);
PartitionCacheKey partKey = new PartitionCacheKey(nameMapping,
values);
HivePartition partition = partitionEntry.getIfPresent(partKey);
if (partition == null) {
+ // Partition metadata cache miss: the exact FileCacheKey
cannot be rebuilt here because it
+ // needs the partition path and input format carried by
HivePartition. Invalidate this
+ // table's cached file listings for the partition by (table id
+ partition values). Scoping
+ // by table id is intentional: matching partition values alone
would also drop other tables'
+ // listings that merely share the same partition value names
(e.g. dt=...) at a different
+ // location, forcing needless re-listing. The exact-key path
below (taken when the partition
+ // is cached) already clears a listing regardless of which
table id populated it.
+ fileEntry.invalidateIf(k -> k.isSameTable(tableId) &&
Objects.equals(k.getPartitionValues(), values));
+ partitionEntry.invalidateKey(partKey);
return;
}
@@ -690,7 +690,7 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
List<String> modifiedPartNames,
List<String> newPartNames) {
for (String partitionName : modifiedPartNames) {
- invalidatePartitionCache(table, partitionName);
+ invalidatePartitionCache(table.getOrBuildNameMapping(),
partitionName);
}
List<String> mergedPartNames =
Lists.newArrayList(modifiedPartNames);
@@ -786,7 +786,7 @@ public class HiveExternalMetaCache extends
AbstractExternalMetaCache {
partitionValuesMap.remove(partitionId);
if (invalidPartitionCache) {
- invalidatePartitionCache(dorisTable, partitionName);
+ invalidatePartitionCache(nameMapping, partitionName);
}
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
index 108a45e0372..151d46252d9 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveMetaStoreCacheTest.java
@@ -26,7 +26,9 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import java.util.ArrayList;
+import java.util.Collections;
import java.util.HashMap;
+import java.util.List;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.atomic.AtomicLong;
@@ -80,6 +82,68 @@ public class HiveMetaStoreCacheTest {
Assertions.assertEquals(0, entrySize(partitionValuesCache));
}
+ @Test
+ public void
testInvalidatePartitionCacheClearsStaleFileCacheOnPartitionMiss() {
+ ThreadPoolExecutor executor =
ThreadPoolManager.newDaemonFixedThreadPool(
+ 1, 1, "refresh", 1, false);
+ ThreadPoolExecutor listExecutor =
ThreadPoolManager.newDaemonFixedThreadPool(
+ 1, 1, "file", 1, false);
+ try {
+ HiveExternalMetaCache cache = new HiveExternalMetaCache(executor,
listExecutor);
+ cache.initCatalog(0, new HashMap<>());
+
+ MetaCacheEntry<HiveExternalMetaCache.FileCacheKey,
HiveExternalMetaCache.FileCacheValue> fileCache =
+ cache.entry(0, HiveExternalMetaCache.ENTRY_FILE,
+ HiveExternalMetaCache.FileCacheKey.class,
+ HiveExternalMetaCache.FileCacheValue.class);
+
+ String dbName = "db";
+ String tbName = "tb";
+ NameMapping nameMapping = NameMapping.createForTest(dbName,
tbName);
+ long catalogId = nameMapping.getCtlId();
+ long tableId = Util.genIdByName(dbName, tbName);
+ long otherTableId = Util.genIdByName(dbName, "tb2");
+
+ String targetPartName = "dt=2024-01-01";
+ List<String> targetValues =
Collections.singletonList("2024-01-01");
+ String otherPartName = "dt=2024-01-02";
+ List<String> otherValues = Collections.singletonList("2024-01-02");
+
+ // Neither the `partition` cache nor the `partition_values` cache
is populated for this table,
+ // simulating entries that were evicted or never loaded.
invalidatePartitionCache must still
+ // clear the stale file listing: it derives the partition values
from the partition name and
+ // cannot rebuild the exact FileCacheKey (which needs the
partition path / input format).
+ HiveExternalMetaCache.FileCacheKey targetFileKey = new
HiveExternalMetaCache.FileCacheKey(
+ catalogId, tableId, "/wh/db/tb/" + targetPartName, "orc",
targetValues);
+ // Same table, a different partition -> must be kept.
+ HiveExternalMetaCache.FileCacheKey otherPartFileKey = new
HiveExternalMetaCache.FileCacheKey(
+ catalogId, tableId, "/wh/db/tb/" + otherPartName, "orc",
otherValues);
+ // A different table that merely shares the same partition value
names at a different location
+ // -> must be kept (the fallback is intentionally scoped by table
id, not by values alone).
+ HiveExternalMetaCache.FileCacheKey otherTableFileKey = new
HiveExternalMetaCache.FileCacheKey(
+ catalogId, otherTableId, "/wh/db/tb2/" + targetPartName,
"orc", targetValues);
+ fileCache.put(targetFileKey, new
HiveExternalMetaCache.FileCacheValue());
+ fileCache.put(otherPartFileKey, new
HiveExternalMetaCache.FileCacheValue());
+ fileCache.put(otherTableFileKey, new
HiveExternalMetaCache.FileCacheValue());
+ Assertions.assertEquals(3, entrySize(fileCache));
+
+ // Partition-level refresh for the target partition. Even though
its `partition` cache entry
+ // is missing, the stale file listing for that partition must
still be invalidated.
+ cache.invalidatePartitionCache(nameMapping, targetPartName);
+
+ Assertions.assertNull(fileCache.getIfPresent(targetFileKey),
+ "stale file cache for the refreshed partition must be
cleared even on partition cache miss");
+ Assertions.assertNotNull(fileCache.getIfPresent(otherPartFileKey),
+ "file cache for other partitions of the same table must
NOT be affected");
+ Assertions.assertNotNull(fileCache.getIfPresent(otherTableFileKey),
+ "file cache for other tables sharing the same partition
values must NOT be affected");
+ Assertions.assertEquals(2, entrySize(fileCache));
+ } finally {
+ executor.shutdownNow();
+ listExecutor.shutdownNow();
+ }
+ }
+
private void putCache(
MetaCacheEntry<HiveExternalMetaCache.FileCacheKey,
HiveExternalMetaCache.FileCacheValue> fileCache,
MetaCacheEntry<HiveExternalMetaCache.PartitionCacheKey,
HivePartition> partitionCache,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
