yujun777 commented on code in PR #69850:
URL: https://github.com/apache/doris/pull/69850#discussion_r4611072290


##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java:
##########
@@ -343,26 +406,162 @@ public BaseTableStream.StreamScanType 
getStreamScanType() {
     }
 
     public Map<Long, Pair<Long, Long>> getPartitionOffsets(List<Long> 
selectedPartitionIds) {
-        return outputUpdateMap.entrySet().stream()
-                .filter(s -> selectedPartitionIds.contains(s.getKey()))
+        return partitionReadStates.entrySet().stream()
+                .filter(s -> selectedPartitionIds.contains(s.getKey()) && 
s.getValue().outputRange != null)
                 .collect(Collectors.toMap(Map.Entry::getKey, s -> {
                     // Storage keeps the real committed TSO points 
(closed-interval semantics).
                     // BE scans a left-closed right-open range [startTso, 
endTso), so convert the
-                    // bounds only in this scan-facing read view. 
outputUpdateMap and the offset
+                    // bounds only in this scan-facing read view. The stored 
output range and the offset
                     // commit path (toOlapTableStreamUpdate) stay on the 
real-TSO coordinate system.
-                    Pair<Long, Long> v = s.getValue();
+                    Pair<Long, Long> v = s.getValue().outputRange;
                     return Pair.of(TSOTimestamp.toExclusiveBound(v.first),
                             TSOTimestamp.toExclusiveBound(v.second));
                 }));
     }
 
     // get history partition offsets partitionId -> (null, 
historicalTimestampOffset)
     public Map<Long, Pair<Long, Long>> getHistoryPartitionOffsets(List<Long> 
selectedPartitionIds) {
-        return outputUpdateMap.entrySet().stream()
-                .filter(s -> selectedPartitionIds.contains(s.getKey()))
+        return partitionReadStates.entrySet().stream()
+                .filter(s -> selectedPartitionIds.contains(s.getKey()) && 
s.getValue().outputRange != null)
                 // historicalTso is an inclusive upper bound; shift to the 
half-open exclusive end.
                 .collect(Collectors.toMap(Map.Entry::getKey,
-                        s -> Pair.of(null, 
TSOTimestamp.toExclusiveBound(s.getValue().first))));
+                        s -> Pair.of(null, 
TSOTimestamp.toExclusiveBound(s.getValue().outputRange.first))));
+    }
+
+    /** The range has already been resolved and TTL-adjusted by the scan 
rewrite. */
+    public Map<Long, Long> estimateChangelogRead(List<Long> partitionIds,
+            Map<Long, Pair<Long, Long>> offsetMap, 
BaseTableStream.StreamScanType scanType) {
+        Pair<Long, String> fallback = 
getTableRowCount(baseTable.getChangelogIndexMeta().getIndexId());

Review Comment:
   Whole-table fallback is now lazy and shared across the selected partitions 
of a scan. StatsCalculator resolves the selected counts first and only fetches 
the ordinary index estimate when a partition is unknown. The FE test verifies 
that known increment and snapshot estimates skip the whole-table lookup and 
that a known pruned scan skips the ordinary index lookup.
   



##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -832,9 +838,14 @@ private void 
executeCommitTxnRequest(CommitTxnRequest.Builder builder, List<Tabl
         long callbackId = 0L;
         StopWatch stopWatch = new StopWatch();
         stopWatch.start();
+        List<TableStreamUpdateInfo> streamUpdates = 
prepareStreamUpdates(builder);
         try {
             txnState = commitTxn(builder, tableList, transactionId, is2PC, 
tabletCommitInfos, tabletIds);
             txnOperated = true;
+            TableStreamManager streamManager = 
Env.getCurrentEnv().getTableStreamManager();
+            for (TableStreamUpdateInfo update : streamUpdates) {
+                streamManager.recordCommittedStreamUpdate(update, 
txnState.getCommitTime());

Review Comment:
   Cloud planning now uses the tuple returned by Meta Service and no longer 
publishes request samples into a local FE cache after commit. The lost-response 
test executes a second commit call after the reported counts change and 
verifies that planning still uses the first accepted sample. The native test 
verifies that the idempotent commit and subsequent read preserve that sample.
   



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java:
##########
@@ -103,30 +125,55 @@ public void installCloudReadStates(Map<Long, 
Cloud.TableStreamPartitionReadState
                         "Cloud Table Stream offset exceeds the statement end 
TSO");
             }
         }
-        cloudReadStates = immutableReadStates;
-        cloudReadStatesInstalled = true;
-        outputUpdateMap.clear();
-        cloudReadStates.forEach((partitionId, state) -> {
-            if (state.getVisibleVersion() > Partition.PARTITION_INIT_VERSION) {
-                Long startTso = state.hasOffsetTso() ? state.getOffsetTso() : 
null;
-                outputUpdateMap.put(partitionId, Pair.of(startTso, 
state.getEndTso()));
-            }
-        });
+        stream.readLock();
+        try {
+            ImmutableMap.Builder<Long, PartitionReadState> snapshots = 
ImmutableMap.builder();
+            immutableReadStates.forEach((id, state) -> {
+                Pair<Long, Long> range = state.getVisibleVersion() > 
Partition.PARTITION_INIT_VERSION
+                        ? Pair.of(state.hasOffsetTso() ? state.getOffsetTso() 
: null, state.getEndTso()) : null;
+                PartitionStreamState consumption = null;
+                if (state.hasOffsetTso()) {
+                    long baseRows = state.hasBaseRowCount() ? 
state.getBaseRowCount() : -1;
+                    long changelogRows = state.hasChangelogRowCount() ? 
state.getChangelogRowCount() : -1;
+                    PartitionStreamState local = stream.getPartitionState(id);
+                    if (local != null && 
Objects.equals(local.getBaselineOffset(), state.getOffsetTso())
+                            && local.getBaseRowCount() == baseRows && 
local.getChangelogRowCount() == changelogRows) {
+                        consumption = local;
+                    } else {
+                        // MS restores the tuple; another FE metadata instance 
has no valid source binding.
+                        boolean history = state.getOffsetState()
+                                == 
Cloud.TableStreamOffsetStatePB.TABLE_STREAM_OFFSET_INITIAL_SNAPSHOT_PENDING;
+                        consumption = PartitionStreamState.restored(

Review Comment:
   Fixed by persisting the source data generation with the offset and row 
counts. FE image recovery and journal replay retain the generation, and a Cloud 
reader validates the tuple returned by Meta Service against its own catalog. 
Partition restore and tablet-set changes invalidate the previous generation. 
The FE tests cover a recovered catalog and a Cloud read with no local stream 
baseline.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to