morningman opened a new issue, #70281: URL: https://github.com/apache/doris/issues/70281
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master @ 72e91896bbdd8760f8bd4e488e5672406b40ce52 (2026-11-03). The code paths below are unchanged on master as of 2026-11-05. ### What's Wrong? In shared-nothing mode, the Table Stream offset is advanced in `updateCatalogAfterCommitted()` when the consuming transaction commits ([DatabaseTransactionMgr.java#L2433-L2446](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L2433-L2446)). The rows it wrote become readable only after publish. If publish is slow or times out, `INSERT INTO ... SELECT ... FROM <stream>` still returns, with the transaction left COMMITTED ([OlapInsertExecutor.java#L236-L251](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/OlapInsertExecutor.java#L236-L251)). A common consumption pattern reads its own target table in the same statement, for example accumulating deltas into a state table with a scheduled job: ```sql INSERT INTO state SELECT d.id, COALESCE(s.amount, 0) + d.delta FROM (SELECT id, SUM(amount) AS delta FROM detail_stream GROUP BY id) d LEFT JOIN state s ON s.id = d.id; ``` If round N is COMMITTED but not yet VISIBLE when round N+1 runs: - round N+1 reads the stream from the advanced offset, so round N's delta is no longer pending; - round N+1 reads `state` without round N's rows; - round N+1's upsert later overwrites round N's result, and round N's delta is lost permanently. IVM already guards against exactly this for aggregate MVs. `IvmDeltaRewriter` fails the refresh with `MV_COMMIT_NOT_VISIBLE` when the previous refresh is committed but not visible, and its comment says that otherwise "that join misses the old rows and the delta is permanently lost" ([IvmDeltaRewriter.java#L76-L90](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/fe/fe-core/src/main/java/org/apache/doris/mtmv/ivm/IvmDeltaRewriter.java#L76-L90)). User-defined stream consumers get no equivalent protection. ### What You Expected? A stream consumer should not observe an advanced offset while the transaction that advanced it is still invisible. Possible approaches: - reject or wait when the stream's previous consumption is COMMITTED but not yet VISIBLE; - make the advanced offset effective only at VISIBLE. At minimum, the Table Stream documentation should warn about read-modify-write consumers. ### How to Reproduce? 1. Create a changelog-enabled detail table, an `append_only` stream on it, and a MoW `state` table. 2. Run the statement above once so that `state` has a baseline. 3. Enable the FE debug point `DatabaseTransactionMgr.finishTransaction.block_visible` (also used by [test_ivm_agg_previous_commit_not_visible.groovy](https://github.com/apache/doris/blob/72e91896bbdd8760f8bd4e488e5672406b40ce52/regression-test/suites/mtmv_p0/ivm/test_ivm_agg_previous_commit_not_visible.groovy#L29)), load a batch into the detail table, and run the statement (round N stays COMMITTED). 4. Load another batch for the same ids and run the statement again (round N+1). 5. Disable the debug point and compare `state` with `SELECT id, SUM(amount) FROM detail GROUP BY id`. Suspected: round N's amounts are missing. ### Anything Else? Found by code reading while documenting Table Stream consumption patterns; not yet reproduced. In shared-storage mode a commit is visible as soon as it succeeds, so that mode should not be affected. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
