CalvinKirs commented on code in PR #66717:
URL: https://github.com/apache/doris/pull/66717#discussion_r3836992905
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonExternalMetaCache.java:
##########
@@ -86,41 +120,324 @@ public Table getPaimonTable(NameMapping nameMapping) {
public PaimonSnapshotCacheValue getSnapshotCache(ExternalTable dorisTable)
{
NameMapping nameMapping = dorisTable.getOrBuildNameMapping();
- return
tableEntry.get(nameMapping.getCtlId()).get(nameMapping).getLatestSnapshotCacheValue();
+ MetaCacheEntry<NameMapping, PaimonTableCacheValue> tables =
tableEntry.get(nameMapping.getCtlId());
+ PaimonTableCacheValue tableValue = tables.get(nameMapping);
+ if (!tables.isEffectivelyEnabled()) {
+ // Projections are keyed by the synthetic generation of a
published table handle. An
+ // ineffective table entry publishes nothing, so nothing keyed by
this load could ever
+ // be looked up again: serve it directly instead of churning the
snapshot entry.
+ PaimonSnapshot fence = loadLatestSnapshotFence(nameMapping,
tableValue).getSnapshot();
+ return executeForGeneration(tableValue, nameMapping,
+ () -> latestSnapshotProjectionLoader.loadAtFence(
+ nameMapping, fence, tableValue.getGeneration()))
+ .bindCapturedAuthenticator(tableValue.getAuthenticator());
+ }
+ LatestFenceOwner owner = new LatestFenceOwner(nameMapping,
tableValue.getGeneration());
+ // Serve the memoized latest projection of this table generation while
it is still
+ // published: the latest read is as stale-until-TTL/refresh as the
cached table
+ // handle itself and costs no snapshot IO, preserving the pre-existing
external
+ // metadata cache contract. The fence is re-observed only when no
projection of this
+ // generation is reachable anymore (first read, expiry, weight
eviction, explicit
+ // invalidation), which is also when rollback ordering below matters.
+ ObservedFence observed = latestObservedFences.get(owner);
+ if (observed != null) {
+ PaimonSnapshotCacheValue memoized =
+
snapshotEntry.get(nameMapping.getCtlId()).peekIfPresent(observed.key);
+ if (memoized != null) {
+ return memoized;
+ }
+ }
+ // Order fence observations, not snapshot ids: a rollback moves the
latest snapshot
+ // backwards, and a concurrent call may finish after a later
observation (reversed
+ // completion). Either way the most recently observed fence is the one
future lookups
+ // read. Capture and number assignment are serialized per owner so the
observation order
+ // always matches the fence-read order; without this, a capture
pausing between the read
+ // and the increment could replace a newer already-published fence
with an older one.
+ PaimonSnapshot fence;
+ long observation;
+ Object captureLock = fenceCaptureLocks.computeIfAbsent(owner, ignored
-> new Object());
Review Comment:
Fixed in the follow-up commit: capture through projection publication now
runs under a finally block that performs the unpublished-generation cleanup on
every path - including the fence-read and projection-load failure points -
conditionally removing the exact registered lock via remove(owner, captureLock)
together with the observed-fence owner. Regression
testFailedFenceOrProjectionLoadsDoNotStrandCaptureLockOwners drives repeated
weight-rejected loads through both exception points and asserts both maps end
empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
