924060929 opened a new pull request, #65334:
URL: https://github.com/apache/doris/pull/65334
### What problem does this PR solve?
Problem Summary:
Partition-level refresh / invalidation
(`HiveExternalMetaCache.invalidatePartitionCache`) returned early when the
target partition was absent from the partition metadata cache — evicted under
`max_hive_partition_cache_num` pressure, or simply never loaded — and left that
partition's file listing cache untouched.
The partition cache (`partition`) and the file listing cache (`file`) are
independent caches with independent size limits (`max_hive_partition_cache_num`
vs `max_external_file_cache_num`) and evict separately, so the partition entry
can be evicted while its file listing survives. In that state a partition-level
refresh cannot rebuild the exact `FileCacheKey` (it needs the partition path /
input format that only the cached `HivePartition` carries), so the old code
gave up and the stale file listing remained. A later query would reload the
partition metadata and re-hit the same stale file listing — i.e. "the partition
exists, but Doris keeps seeing the old files/data".
Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a
predicate-based invalidation on the file cache keyed by `table id + partition
values`, so the stale listing for that partition is cleared without
over-invalidating other partitions. This mirrors table-level invalidation,
which already invalidates the file cache by predicate.
The partition invalidation coordinator is also decoupled from
`ExternalTable` to `NameMapping` (mirroring the existing
`invalidateTableCache(NameMapping)`) so the path can be unit tested directly.
### Release note
Fix a Hive external catalog consistency issue where a partition-level
refresh could leave a stale file listing cached when the partition metadata
cache had been evicted, causing queries to keep seeing outdated files for that
partition.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test
- [ ] No need to test
- Behavior changed:
- [ ] No.
- [x] Yes. Partition-level refresh now also invalidates the file listing
cache when the partition metadata cache entry is missing (previously it was
left stale).
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
