englefly commented on code in PR #69850:
URL: https://github.com/apache/doris/pull/69850#discussion_r4608619645


##########
fe/fe-core/src/main/java/org/apache/doris/cloud/transaction/CloudGlobalTransactionMgr.java:
##########
@@ -832,9 +838,14 @@ private void 
executeCommitTxnRequest(CommitTxnRequest.Builder builder, List<Tabl
         long callbackId = 0L;
         StopWatch stopWatch = new StopWatch();
         stopWatch.start();
+        List<TableStreamUpdateInfo> streamUpdates = 
prepareStreamUpdates(builder);
         try {
             txnState = commitTxn(builder, tableList, transactionId, is2PC, 
tabletCommitInfos, tabletIds);
             txnOperated = true;
+            TableStreamManager streamManager = 
Env.getCurrentEnv().getTableStreamManager();
+            for (TableStreamUpdateInfo update : streamUpdates) {
+                streamManager.recordCommittedStreamUpdate(update, 
txnState.getCommitTime());

Review Comment:
   **[P2] Cloud 提交重试应发布 Meta Service 实际接受的计数**
   
   如果首次提交已成功但响应丢失,FE 在 tablet 上报更新后发起新的提交尝试时会重新采样。Meta Service 对已可见事务返回 `OK` 
并保留首次提交的计数,这里却发布了第二次采样。随后读取会因两组计数不同而拒绝本地基线,同偏移量的发布保护也会阻止替换。建议使用 Meta Service 
实际接受的元组,或在提交重试间复用原始采样,并覆盖响应丢失后计数变化的场景。



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java:
##########
@@ -103,30 +125,55 @@ public void installCloudReadStates(Map<Long, 
Cloud.TableStreamPartitionReadState
                         "Cloud Table Stream offset exceeds the statement end 
TSO");
             }
         }
-        cloudReadStates = immutableReadStates;
-        cloudReadStatesInstalled = true;
-        outputUpdateMap.clear();
-        cloudReadStates.forEach((partitionId, state) -> {
-            if (state.getVisibleVersion() > Partition.PARTITION_INIT_VERSION) {
-                Long startTso = state.hasOffsetTso() ? state.getOffsetTso() : 
null;
-                outputUpdateMap.put(partitionId, Pair.of(startTso, 
state.getEndTso()));
-            }
-        });
+        stream.readLock();
+        try {
+            ImmutableMap.Builder<Long, PartitionReadState> snapshots = 
ImmutableMap.builder();
+            immutableReadStates.forEach((id, state) -> {
+                Pair<Long, Long> range = state.getVisibleVersion() > 
Partition.PARTITION_INIT_VERSION
+                        ? Pair.of(state.hasOffsetTso() ? state.getOffsetTso() 
: null, state.getEndTso()) : null;
+                PartitionStreamState consumption = null;
+                if (state.hasOffsetTso()) {
+                    long baseRows = state.hasBaseRowCount() ? 
state.getBaseRowCount() : -1;
+                    long changelogRows = state.hasChangelogRowCount() ? 
state.getChangelogRowCount() : -1;
+                    PartitionStreamState local = stream.getPartitionState(id);
+                    if (local != null && 
Objects.equals(local.getBaselineOffset(), state.getOffsetTso())
+                            && local.getBaseRowCount() == baseRows && 
local.getChangelogRowCount() == changelogRows) {
+                        consumption = local;
+                    } else {
+                        // MS restores the tuple; another FE metadata instance 
has no valid source binding.
+                        boolean history = state.getOffsetState()
+                                == 
Cloud.TableStreamOffsetStatePB.TABLE_STREAM_OFFSET_INITIAL_SNAPSHOT_PENDING;
+                        consumption = PartitionStreamState.restored(

Review Comment:
   **[P2] 让恢复后的 FE 能使用已持久化的基线**
   
   当前 FE 没有匹配的本地采样时,`installCloudReadStates` 会从 Meta Service 恢复计数,但恢复出的状态没有 
`sourceEpoch`,因此 `matches()` 始终拒绝它。存算一体模式的日志回放也会清除该绑定。即使在提交所在的 
FE,并发读持有流锁也会让零等待发布跳过更新。这些路径会让少量未消费变更回退到整个保留的 changelog 
行数,直到下一次消费推进偏移量。建议持久化可在恢复后和跨 FE 校验的源数据版本标识,并测试从恢复状态生成的执行计划。



##########
fe/fe-core/src/main/java/org/apache/doris/catalog/stream/OlapTableStreamWrapper.java:
##########
@@ -343,26 +406,162 @@ public BaseTableStream.StreamScanType 
getStreamScanType() {
     }
 
     public Map<Long, Pair<Long, Long>> getPartitionOffsets(List<Long> 
selectedPartitionIds) {
-        return outputUpdateMap.entrySet().stream()
-                .filter(s -> selectedPartitionIds.contains(s.getKey()))
+        return partitionReadStates.entrySet().stream()
+                .filter(s -> selectedPartitionIds.contains(s.getKey()) && 
s.getValue().outputRange != null)
                 .collect(Collectors.toMap(Map.Entry::getKey, s -> {
                     // Storage keeps the real committed TSO points 
(closed-interval semantics).
                     // BE scans a left-closed right-open range [startTso, 
endTso), so convert the
-                    // bounds only in this scan-facing read view. 
outputUpdateMap and the offset
+                    // bounds only in this scan-facing read view. The stored 
output range and the offset
                     // commit path (toOlapTableStreamUpdate) stay on the 
real-TSO coordinate system.
-                    Pair<Long, Long> v = s.getValue();
+                    Pair<Long, Long> v = s.getValue().outputRange;
                     return Pair.of(TSOTimestamp.toExclusiveBound(v.first),
                             TSOTimestamp.toExclusiveBound(v.second));
                 }));
     }
 
     // get history partition offsets partitionId -> (null, 
historicalTimestampOffset)
     public Map<Long, Pair<Long, Long>> getHistoryPartitionOffsets(List<Long> 
selectedPartitionIds) {
-        return outputUpdateMap.entrySet().stream()
-                .filter(s -> selectedPartitionIds.contains(s.getKey()))
+        return partitionReadStates.entrySet().stream()
+                .filter(s -> selectedPartitionIds.contains(s.getKey()) && 
s.getValue().outputRange != null)
                 // historicalTso is an inclusive upper bound; shift to the 
half-open exclusive end.
                 .collect(Collectors.toMap(Map.Entry::getKey,
-                        s -> Pair.of(null, 
TSOTimestamp.toExclusiveBound(s.getValue().first))));
+                        s -> Pair.of(null, 
TSOTimestamp.toExclusiveBound(s.getValue().outputRange.first))));
+    }
+
+    /** The range has already been resolved and TTL-adjusted by the scan 
rewrite. */
+    public Map<Long, Long> estimateChangelogRead(List<Long> partitionIds,
+            Map<Long, Pair<Long, Long>> offsetMap, 
BaseTableStream.StreamScanType scanType) {
+        Pair<Long, String> fallback = 
getTableRowCount(baseTable.getChangelogIndexMeta().getIndexId());

Review Comment:
   **[P2] 推迟计算全表回退值**
   
   即使只选中一个分区,且该分区已有可用的基线差值,`estimateChangelogRead` 也先遍历源表所有分区汇总 changelog 
行数,必要时还会汇总基表行数。快照路径也提前汇总基表,`StatsCalculator` 
在合并显式估算前又执行全索引查询。建议仅在选中分区确实需要回退时才计算全表行数,并复用该结果。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to