yujun777 commented on code in PR #69850:
URL: https://github.com/apache/doris/pull/69850#discussion_r4611072290
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java:
##########
@@ -343,26 +406,162 @@ public BaseTableStream.StreamScanType
getStreamScanType() {
}
public Map<Long, Pair<Long, Long>> getPartitionOffsets(List<Long>
selectedPartitionIds) {
- return outputUpdateMap.entrySet().stream()
- .filter(s -> selectedPartitionIds.contains(s.getKey()))
+ return partitionReadStates.entrySet().stream()
+ .filter(s -> selectedPartitionIds.contains(s.getKey()) &&
s.getValue().outputRange != null)
.collect(Collectors.toMap(Map.Entry::getKey, s -> {
// Storage keeps the real committed TSO points
(closed-interval semantics).
// BE scans a left-closed right-open range [startTso,
endTso), so convert the
- // bounds only in this scan-facing read view.
outputUpdateMap and the offset
+ // bounds only in this scan-facing read view. The stored
output range and the offset
// commit path (toOlapTableStreamUpdate) stay on the
real-TSO coordinate system.
- Pair<Long, Long> v = s.getValue();
+ Pair<Long, Long> v = s.getValue().outputRange;
return Pair.of(TSOTimestamp.toExclusiveBound(v.first),
TSOTimestamp.toExclusiveBound(v.second));
}));
}
// get history partition offsets partitionId -> (null,
historicalTimestampOffset)
public Map<Long, Pair<Long, Long>> getHistoryPartitionOffsets(List<Long>
selectedPartitionIds) {
- return outputUpdateMap.entrySet().stream()
- .filter(s -> selectedPartitionIds.contains(s.getKey()))
+ return partitionReadStates.entrySet().stream()
+ .filter(s -> selectedPartitionIds.contains(s.getKey()) &&
s.getValue().outputRange != null)
// historicalTso is an inclusive upper bound; shift to the
half-open exclusive end.
.collect(Collectors.toMap(Map.Entry::getKey,
- s -> Pair.of(null,
TSOTimestamp.toExclusiveBound(s.getValue().first))));
+ s -> Pair.of(null,
TSOTimestamp.toExclusiveBound(s.getValue().outputRange.first))));
+ }
+
+ /** The range has already been resolved and TTL-adjusted by the scan
rewrite. */
+ public Map<Long, Long> estimateChangelogRead(List<Long> partitionIds,
+ Map<Long, Pair<Long, Long>> offsetMap,
BaseTableStream.StreamScanType scanType) {
+ Pair<Long, String> fallback =
getTableRowCount(baseTable.getChangelogIndexMeta().getIndexId());
Review Comment:
Whole-table fallback is now lazy and shared across the selected partitions
of a scan. StatsCalculator resolves the selected counts first and only fetches
the ordinary index estimate when a partition is unknown. The FE test verifies
that known increment and snapshot estimates skip the whole-table lookup and
that a known pruned scan skips the ordinary index lookup.
##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -832,9 +838,14 @@ private void
executeCommitTxnRequest(CommitTxnRequest.Builder builder, List<Tabl
long callbackId = 0L;
StopWatch stopWatch = new StopWatch();
stopWatch.start();
+ List<TableStreamUpdateInfo> streamUpdates =
prepareStreamUpdates(builder);
try {
txnState = commitTxn(builder, tableList, transactionId, is2PC,
tabletCommitInfos, tabletIds);
txnOperated = true;
+ TableStreamManager streamManager =
Env.getCurrentEnv().getTableStreamManager();
+ for (TableStreamUpdateInfo update : streamUpdates) {
+ streamManager.recordCommittedStreamUpdate(update,
txnState.getCommitTime());
Review Comment:
Cloud planning now uses the tuple returned by Meta Service and no longer
publishes request samples into a local FE cache after commit. The lost-response
test executes a second commit call after the reported counts change and
verifies that planning still uses the first accepted sample. The native test
verifies that the idempotent commit and subsequent read preserve that sample.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java:
##########
@@ -103,30 +125,55 @@ public void installCloudReadStates(Map<Long,
Cloud.TableStreamPartitionReadState
"Cloud Table Stream offset exceeds the statement end
TSO");
}
}
- cloudReadStates = immutableReadStates;
- cloudReadStatesInstalled = true;
- outputUpdateMap.clear();
- cloudReadStates.forEach((partitionId, state) -> {
- if (state.getVisibleVersion() > Partition.PARTITION_INIT_VERSION) {
- Long startTso = state.hasOffsetTso() ? state.getOffsetTso() :
null;
- outputUpdateMap.put(partitionId, Pair.of(startTso,
state.getEndTso()));
- }
- });
+ stream.readLock();
+ try {
+ ImmutableMap.Builder<Long, PartitionReadState> snapshots =
ImmutableMap.builder();
+ immutableReadStates.forEach((id, state) -> {
+ Pair<Long, Long> range = state.getVisibleVersion() >
Partition.PARTITION_INIT_VERSION
+ ? Pair.of(state.hasOffsetTso() ? state.getOffsetTso()
: null, state.getEndTso()) : null;
+ PartitionStreamState consumption = null;
+ if (state.hasOffsetTso()) {
+ long baseRows = state.hasBaseRowCount() ?
state.getBaseRowCount() : -1;
+ long changelogRows = state.hasChangelogRowCount() ?
state.getChangelogRowCount() : -1;
+ PartitionStreamState local = stream.getPartitionState(id);
+ if (local != null &&
Objects.equals(local.getBaselineOffset(), state.getOffsetTso())
+ && local.getBaseRowCount() == baseRows &&
local.getChangelogRowCount() == changelogRows) {
+ consumption = local;
+ } else {
+ // MS restores the tuple; another FE metadata instance
has no valid source binding.
+ boolean history = state.getOffsetState()
+ ==
Cloud.TableStreamOffsetStatePB.TABLE_STREAM_OFFSET_INITIAL_SNAPSHOT_PENDING;
+ consumption = PartitionStreamState.restored(
Review Comment:
Fixed by persisting the source data generation with the offset and row
counts. FE image recovery and journal replay retain the generation, and a Cloud
reader validates the tuple returned by Meta Service against its own catalog.
Partition restore and tablet-set changes invalidate the previous generation.
The FE tests cover a recovered catalog and a Cloud read with no local stream
baseline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]