wenzhenghu opened a new issue, #66001:
URL: https://github.com/apache/doris/issues/66001
### Background
This issue tracks a metadata cache correctness gap around the ordering
between object entry publication/invalidation and ID-to-name navigation state
publication.
In the current code structure, this is represented by:
- `ExternalCatalog`: `dbIdToName` vs `databases`
- `ExternalDatabase`: `tableIdToName` vs `tables`
This issue is related to PR #65126, because the current PR makes the problem
more explicit on the hot-hit path. However, the underlying class of concurrency
problem is not entirely new: similar ordering gaps already existed in the
baseline implementation on miss-load / cache-update paths before this refactor.
### Problem
Named lookup may update `dbIdToName` / `tableIdToName` after the object
entry has already been invalidated concurrently.
A minimal example on the table path:
1. Thread A executes `getTableNullable("orders")` and gets a hot object from
`tables.get("orders")`
2. Thread B concurrently executes DROP / RENAME and invalidates both the
object entry and the corresponding `tableIdToName`
3. Thread A resumes and writes `tableIdToName.put(tableId, "orders")`
This restores stale by-ID navigation after cleanup has already removed the
object.
The same pattern also applies to miss-load paths when object publication is
rejected after a generation change but the loaded object is still used to
republish ID navigation state outside the object entry's publication protocol.
Database-level lookup has the same issue with `dbIdToName` and `databases`.
### Current impact assessment
This is a real correctness issue, but it currently looks like a
low-probability race window rather than a common user-facing failure mode.
Triggering it requires a specific interleaving between lookup and concurrent
invalidation / drop / rename. Based on the current reachable code paths, the
practical impact also seems limited:
- the table-level path is mainly reachable through relatively narrow by-ID
resolution flows
- the database-level path is somewhat more reachable, but still not part of
the hottest user-facing execution paths
- in the currently understood cases, the effect is more likely to be stale
by-ID navigation or repeated reload attempts after cleanup, rather than a broad
correctness failure on common query paths
### Why this is tracked separately
For PR #65126 specifically:
- the current refactor introduces a new hot-hit window where named lookup
republishes ID navigation state after reading a hot object
- at the same time, the broader ordering problem is not purely introduced by
this PR, because baseline code already had similar miss-load / cache-update
ordering gaps
Given the relatively narrow trigger window and limited current impact, this
is better tracked as a dedicated follow-up issue rather than expanding PR
#65126 into a broader cache linearization rework late in the review cycle.
### Follow-up scope
The follow-up should cover both:
1. Current code structure
- `dbIdToName` vs `databases`
- `tableIdToName` vs `tables`
2. Relevant publication paths
- hot named lookup
- miss-load / manual load
- incremental cache update
- object invalidation / unregister / rename / drop
### Expected fix direction
The fix should ensure that ID-map updates are linearized with successful
object publication and invalidation, instead of being updated as a separate
late side effect.
At a minimum, the final design should guarantee:
- failed or generation-rejected object publication must not restore stale ID
navigation
- invalidate / unregister must not be undone by a concurrent late ID-map
write
- by-ID lookup must not retain stale navigation after object cleanup
### Test expectations
Please add deterministic concurrency tests for both database and table
paths, including at least:
- hot-hit vs drop/rename
- miss-load vs drop/rename
- generation-rejected load vs invalidation
### Related context
- PR #65126
- Review discussion:
https://github.com/apache/doris/pull/65126#discussion_r3642519993
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
