924060929 commented on code in PR #65334:
URL: https://github.com/apache/doris/pull/65334#discussion_r3536423853
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java:
##########
@@ -641,6 +644,13 @@ private void invalidatePartitionCache(ExternalTable
dorisTable, String partition
PartitionCacheKey partKey = new PartitionCacheKey(nameMapping,
values);
HivePartition partition = partitionEntry.getIfPresent(partKey);
if (partition == null) {
+ // Partition metadata cache miss: the exact FileCacheKey
cannot be rebuilt here because it
+ // needs the partition path and input format carried by
HivePartition. A stale file listing
+ // for this partition may still exist though, because the file
cache and partition cache are
+ // independent and evict separately. Fall back to a
predicate-based invalidation keyed by
Review Comment:
**Partition-values entry evicted — fixed.** The fallback no longer reads the
partition-values cache: `values` is now derived directly from the partition
name via `HiveUtil.toPartitionValues(partitionName)` (the same call that
populates that cache), and the two early returns on a missing partition-values
entry are removed. Invalidation therefore runs even when
`ENTRY_PARTITION_VALUES` was evicted while `ENTRY_FILE` is still populated. The
unit test no longer populates the partition-values cache, covering exactly that
state.
**`isSameTable(tableId)` scope — kept, with reasoning.** An
identity-consistent match isn't reachable in this branch: `FileCacheKey` reuse
identity is `(catalogId, location, inputFormat, partitionValues)`, but the
branch is entered precisely because the `HivePartition` — the only source of
`location`/`inputFormat` — is a cache miss, so both are unavailable. Matching
on `partitionValues` alone would over-invalidate: partition value names such as
`dt=2024-01-01` are shared by many unrelated tables at different locations, so
a single partition refresh would evict all of their listings and force needless
re-listing. Scoping by `table id + partition values` precisely clears this
table's own listings for the partition (the common case).
The residual gap — a listing populated by a *different* table that shares
the exact same location+format+values — requires two tables to physically
overlap partition directories *and* a partition-cache miss on this table, and
it is already handled by the exact-key path (which ignores the owning table id
via `FileCacheKey.equals`) whenever the partition metadata is cached. I added a
test asserting that a different table's same-value listing is intentionally
preserved (so we don't thrash unrelated tables). If you'd prefer conservatism
over the re-listing cost, I can switch to a coarse table-level fallback —
flagging the tradeoff rather than silently choosing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
