morningman opened a new issue, #65418:
URL: https://github.com/apache/doris/issues/65418
### Description
This issue tracks the end-to-end incremental computation capability in
Apache Doris.
The implementation is being built in three layers:
```text
OLAP table DML
-> row binlog: durable row-level changes, before images, operation type,
LSN/TSO
-> Table Stream: change-query semantics, per-partition consumption
offsets, snapshots and reset
-> MTMV incremental maintenance: derive and transactionally apply only
the affected result changes
```
This tracker replaces the stale umbrella issue #57921 and incorporates the
row-binlog foundation originally tracked by #61956.
### Goals
- Provide a durable row-level change log for Doris internal tables.
- Expose deterministic incremental consumption through Table Stream.
- Provide snapshot reads aligned with stream consumption points for joins
and recovery.
- Integrate Table Stream with MTMV planning and refresh execution for
row-level incremental maintenance.
- Preserve correctness across replicas, compaction, schema changes,
concurrent consumers, failures, retries, and restart/replay.
- Add complete unit, regression, compatibility, and failure-recovery
coverage.
- Document supported table models, SQL syntax, semantics, limitations, and
operational guidance.
### Scope boundary
This tracker covers the internal row-binlog and Table Stream stack used by
incremental computation, plus its MTMV integration. External CDC Streaming Jobs
and general MTMV partition refresh or query-rewrite changes are out of scope
unless they directly modify this stack.
FE-side (CCR) binlog is listed under "Adjacent work" below: it is a separate
feature, but several correctness issues tracked here live in that code and are
reported against the same stack.
The repository review has been refreshed through **2026-09-01**. PR #61382
predates the original review window but is included because it is the Table
Stream metadata and DDL foundation.
### Status summary (2026-09-01)
| Layer | Merged | Open / Draft | Assessment |
| --- | --- | --- | --- |
| Row binlog | 23 | 7 | Main path complete; remaining work is scheduling
locality, partial update, and throughput. |
| Table Stream | 17 | 6 | Local and cloud paths complete; metadata lifecycle
and DDL surface still have gaps. |
| MTMV incremental maintenance | 0 | 1 draft | Implemented on a branch, but
nothing has been merged. **This is the critical path.** |
### Progress
#### Row binlog foundation
Merged — core path:
- [x] #62058 — define row-binlog metadata and schema.
- [x] #63110 — implement row-binlog read and write paths.
- [x] #63643 — implement replica clone, scheduling, and compaction.
- [x] #64133 — allocate LSNs at the sink and keep LSN/TSO semantics
consistent across replicas.
- [x] #65810 — support cloud binlog and an independent binlog tablet.
- [x] #66883 — derive row binlog in the segment-writer transform chain and
delete `RowBinlogSegmentWriter`. Supersedes #64674.
- [x] #66889 — add a per-row LSN column for detail (DUP) tables, required by
MTMV incremental maintenance.
Merged — fixes:
- [x] #64249, #65032 — cumulative policy NPE during binlog scheduling and
binlog compaction.
- [x] #64545, #65663, #65686 — incomplete binlog metadata in the grouped
rowset writer and its tests.
- [x] #65076 — include hidden key columns while excluding hidden non-key
internal columns.
- [x] #65438 — align the row-binlog AFTER schema with before values.
- [x] #65481 — report missing row binlog for INCR queries.
- [x] #66046 — emit INSERT rather than UPDATE when an insert follows a
delete.
- [x] #66093 — disallow Row Binlog on MOW tables with cluster keys.
- [x] #66432 — fix the BE crash when an APPEND_ONLY row-binlog scan omits
key columns.
- [x] #66553 — wrap Row Binlog AFTER columns as nullable before conversion.
- [x] #66601 — support complex `array_agg` state serialization.
- [x] #66648 — avoid mutating shared source columns during concurrent Row
Binlog segment flushes.
- [x] #66665 — fix row-binlog routing for auto partitions.
- [x] #66840 — allow key-only partial updates with row binlog.
Open:
- [ ] #66191 — keep row binlog tablets colocated with base tablets. Approved
by five reviewers but not merged; #67092 and #67106 are correctness issues on
this path.
- [ ] #66899 vs #66764 — flexible partial update. #66899 adds support,
#66764 rejects it at DDL time. **These are mutually exclusive and a decision is
needed** before #65866 can be closed.
- [ ] #67050 — resolve before columns by unique ID.
- [ ] #66963 — restrict state types to aggregate tables (HLL/QuantileState
path behind #65865).
- [ ] #64484, #64495 — missing binlog column index, and timestamp nullable
mismatch. Open since mid-June; need confirmation of whether #66883 already
covers them.
- [ ] #65947 — enabling binlog reduces import throughput by about 65%. The
only attempted fix (#66894) was closed; **no owner and no current fix**. This
blocks enabling the feature by default.
#### Table Stream foundation
Merged — core path:
- [x] #61382 — add stream metadata, DDL, and information-schema visibility.
- [x] #62453 — add stream query and transactional consumption infrastructure.
- [x] #63850 — connect OLAP Table Stream to row binlog and add DETAIL,
APPEND_ONLY, and MIN_DELTA scan modes.
- [x] #64776 — add stream snapshot/reset reads, table incremental reads, and
time-travel snapshot support.
- [x] #65859 — add the Cloud Table Stream control plane.
- [x] #66743 — add comprehensive Cloud Table Stream coverage.
Merged — fixes:
- [x] #65468, #65657 — stream scan normalization, snapshot/reset pruning,
and partition-prune state propagation.
- [x] #65636 — rename the misleading `StreamTableInfo`.
- [x] #65826 — MIN_DELTA delete op returns the first op's before image.
- [x] #65973 — force binlog TSO predicate pushdown and forbid other
pushdowns.
- [x] #66338 — fix multi-segment row-binlog reads for MoW tables.
- [x] #66683 — parse stream properties before base-table validation so valid
APPEND_ONLY and DETAIL streams are not rejected.
- [x] #66739, #67079 — localize Cloud Table Stream rewrite state handling
and its test hook.
- [x] #67126 — commit table stream offsets for empty inserts.
- [x] #67173 — remove the base-table cache and use a dynamic schema.
Open:
- [ ] #65274 — expand Table Stream negative and exception-path regression
coverage. Draft since 2026-07-06; without it the negative paths have no
regression protection.
- [ ] #66287 and #65698 — both mark a stream stale/unavailable after its
base table is dropped (#65389). **Duplicate effort; one needs to be picked.**
Note: #66287 was previously checked off in #65265 but is still a draft.
- [ ] #65959 — fix table stream change-type predicate pushdown.
- [ ] #67130 — derive table stream schema from the base table (#67109).
- [ ] #67181 — wait for running transactions before incremental reads in
cloud mode.
- [ ] #65388, #65390 — `ALTER STREAM SET COMMENT` and `CANCEL ALTER STREAM`
are not supported. No owner.
#### MTMV incremental maintenance
**Corrected.** The previous revision of this tracker stated that no MTMV
implementation PR had been identified. That is no longer accurate:
- #66719 is the focused tracker for this layer.
- #62606 is the implementation PR. It has been open since 2026-04-20 and is
**still a draft: +47922 / -889 across 343 files and 273 commits, with no review
and no approval.** #66266 and #61496 are pipeline-test drafts from the same
branch.
Per #66719, the branch already covers projection, filter, aggregate
(COUNT/SUM/AVG/MIN/MAX, bitmap aggregates, expression arguments, bare GROUP
BY), INNER/CROSS/LEFT OUTER/FULL OUTER joins, UNION ALL, subquery aliases,
`REFRESH MATERIALIZED VIEW ... INCREMENTAL/PARTITIONS`, dry-run, automatic
full-refresh fallback, chained IVM MTMV, `ivm_use_full_keys`, and regression
tests under `mtmv_p0/ivm`.
The blocker is not implementation, it is **reviewability and landing**. A
single PR of this size cannot be reviewed effectively, and this layer currently
has no assigned reviewer.
- [ ] Split #62606 into independently mergeable PRs (delta plan derivation,
per-operator incremental semantics, MTMV lifecycle integration, atomic
result-apply plus offset advancement, fallback and observability, tests).
- [ ] Assign FE and storage reviewers for the split PRs.
- [ ] Define supported MTMV query shapes and eligibility/fallback rules.
- [ ] Derive delta plans from Table Streams, including snapshot sides for
joins.
- [ ] Define incremental semantics for projection, filter, join, union,
aggregation, and supported expressions.
- [ ] Integrate stream creation and lifecycle with MTMV creation,
alteration, pause/resume, and drop.
- [ ] Execute result changes and stream-offset advancement atomically.
- [ ] Support full-refresh bootstrap/reset and fallback when incremental
maintenance is not valid.
- [ ] Define retry, idempotency, concurrent refresh, failover, and EditLog
replay behavior.
- [ ] Add observability for refresh mode, consumed offsets, lag, fallback
reason, and failures.
- [ ] Add FE and BE unit tests plus end-to-end regression tests for
correctness and recovery.
- [ ] Add user and operator documentation.
#### Adjacent work
- #61198 — global monotonically increasing timestamp service (TSO). Closed.
- #65978 / #66191 — row binlog tablet locality during scheduling.
- #66930 — preserve binlog config across REPLACE TABLE (#66571, #67105).
- #65484 — cover binlog config validation routing (#65383).
- #67156 — single replica ingest binlog.
- #63134 — snapshot creation fails with a hard-link NOT_FOUND error when
binlog is enabled during a CLONE task.
### Bug and correctness backlog
#65265 is the consolidated bug and test backlog. Current state:
- Bug list: 7 of 17 closed.
- Correctness issues: 3 of 22 closed (#66273, #66338, #67095). #66287 is
checked off there but is still a draft PR.
- **17 correctness issues have no linked PR and no assignee.**
The unowned correctness issues (#67091–#67094, #67096–#67108) were filed
together on 2026-08-25 and are design-level defects rather than isolated bugs:
BE restart recovery, replica repair and balancing, TSO boundary semantics,
cloud publish/commit failure handling, leader/follower metadata divergence, and
lock/scan target disagreement after RENAME or REPLACE TABLE. They need owners
before they can be scheduled.
### Correctness and compatibility checklist
- [ ] DUP and supported UNIQUE KEY MOW table semantics are documented and
tested.
- [ ] Full update, partial update, delete, sequence column, hidden key, and
schema-change paths are covered.
- [ ] LSN/TSO ordering is deterministic across replicas and remains valid
after clone and compaction.
- [ ] Concurrent stream consumption either commits data and offsets together
or fails without advancing offsets.
- [ ] Snapshot, reset, and incremental reads agree at partition boundaries
and after restart.
- [ ] Binlog retention cannot silently invalidate an active stream or MTMV;
stale state is observable and recoverable.
- [ ] Unsupported plans fail clearly or use a documented full-refresh
fallback.
- [ ] Cloud-mode support or exclusion is explicit.
- [ ] Upgrade, downgrade, rolling restart, backup/restore, and replication
behavior is defined before general availability.
- [ ] Enabling binlog does not regress import throughput to an unacceptable
degree (#65947).
### Overlapping or superseded PRs
- #64137 — closed; its compaction-policy crash path is covered by #64249.
- #64484 and #64495 — overlap with the metadata and nullable-column fixes
merged in #64545; needs confirmation against #66883.
- #64674 — closed; superseded by #66883.
- #66664 — closed; the HLL/state-type path is now handled by #66963.
- #66894 — closed; #65947 remains unresolved.
### Exit criteria
- Row binlog and Table Stream semantics are stable and documented.
- At least one supported MTMV query class refreshes incrementally end to end
with atomic offset advancement.
- Correctness, failure recovery, replay, compaction, and schema evolution
tests pass in the supported deployment modes.
- Unsupported cases have explicit diagnostics and a documented fallback.
- Remaining limitations and follow-up work are tracked by linked issues or
PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
