wenzhenghu opened a new issue, #65779:
URL: https://github.com/apache/doris/issues/65779
### Background
This issue is a follow-up for PR #65126.
PR #65126 refactors the external metadata names cache to use immutable
snapshots with copy-on-write style updates. That change was introduced for
correctness rather than performance.
The main goal of this refactor is to make name-cache publication and
invalidation semantics safe under concurrent loads and incremental HMS events.
In particular, the PR needs a names-cache representation that can:
- publish a consistent snapshot to readers
- avoid in-place mutation of shared cache state
- cooperate with generation-based invalidation / stale-load fencing
- prevent lagging async/manual loads from overwriting newer HMS event updates
Using immutable snapshots helps make these correctness guarantees explicit
and avoids readers observing partially updated name/index state during
concurrent cache refresh or event handling.
At the same time, this refactor also changes the HMS event update path from
the previous incremental-style update to a full snapshot rebuild in some
hot-name-cache cases. As a result, it introduces a real performance regression
on HMS CREATE/DROP event handling for very large metadata sets.
This follow-up is only relevant if PR #65126 is merged.
### Problem
On HMS table/database name events, the current update path may rebuild the
full names snapshot for a single name change. In large external catalogs, one
CREATE or DROP event may:
- scan the full names list
- copy the full list
- recreate existing entries
- rebuild derived indexes and maps
This makes a single HMS event cost O(n) in the size of the hot names
snapshot.
### Impact
This is a performance regression introduced by PR #65126, not a correctness
blocker.
The impact is most visible on the HMS event path with very large metadata
sets. In normal usage, CREATE/DROP is usually not a high-frequency operation,
and most users will not have extremely large numbers of databases or tables
under a single catalog/database. The problem is therefore concentrated in
large-scale HMS event scenarios.
Also, this copy-on-write overhead has already been discussed in earlier
review rounds, and parts of the PR series have already reduced the impact on
more frequent execution paths. The remaining concern is mainly the HMS event
path.
### Why this is tracked separately
By the current review stage of PR #65126, the remaining issues are already
being narrowed down progressively. Fixing this regression properly likely
requires a broader redesign of the names-cache update/publication strategy,
which would significantly expand scope and increase late-stage rework risk.
So instead of mixing that architectural change into the current PR, this
issue tracks the follow-up performance fix after PR #65126 is merged, or
together with other known HMS event follow-up work.
### Candidate directions
Possible directions under consideration:
1. **Stripe/shard the names cache**
- split the names snapshot/index into multiple stripes
- limit copy/rebuild work to the affected subset for each event
- preserve the current snapshot visibility guarantees
2. **Control deep-copy vs reference reuse**
- add an internal mechanism to choose whether immutable entries are
deep-copied or directly reused
- reduce redundant object recreation and index rebuild work
- keep publication safety and immutability guarantees explicit
These are candidate ideas only and still need design validation.
### Follow-up expectations
The follow-up should:
- reduce the O(n) rebuild cost for ordinary single-name HMS events
- preserve concurrency and snapshot correctness
- include targeted benchmark or regression coverage for large hot snapshots
and event bursts
- clearly document the invariants of the final cache design
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
