Yukang-Lian opened a new pull request, #65859:
URL: https://github.com/apache/doris/pull/65859
### What problem does this PR solve?
Issue Number: None
Related PR: #65810
Problem Summary:
Table Stream already provides incremental consumption in storage-compute
integrated deployments, where FE owns each partition offset. In Cloud mode, the
target-table commit is finalized by MetaService, so keeping offsets in FE
cannot atomically commit target data and advance Stream offsets.
This PR adds the Cloud Table Stream control plane:
- models a Stream as a typed Index and reuses the existing prepare-index,
commit-partition, commit-index, Drop Index Log, Recycle Index, and
InstanceRecycler lifecycle;
- stores Latest and Versioned per-partition offsets in MetaService and
resolves one statement-level read snapshot, including clone-chain inheritance;
- performs expected-offset CAS and target-table commit in the same immediate
FDB transaction, with no lazy-commit fallback;
- integrates Cloud create, query, consumption, FORCE drop, consumption
status, capability gating, metadata modes, GC, consistency checks, and
operational key tooling into FE and MetaService;
- preserves the existing non-Cloud Table Stream path and ensures MV
temporary rewriting neither prunes unresolved Cloud Stream partitions nor
issues a second read-state RPC.
This draft depends on #65810 for the Cloud Row Binlog data plane. Before
Cloud E2E is enabled, the integration must confirm that every visible
partition, including an empty partition and a newly added partition, has an
authoritative `VersionPB.visible_tso` and empty-boundary TSO. The control plane
intentionally fails when that metadata is missing instead of treating metadata
loss as an empty partition.
Operational compatibility note: `fdb_kv_ranges_count` adds a `key_space`
label so Latest (`0x01`) and Versioned (`0x03`) keys are counted separately.
Existing monitoring queries and dashboards must be updated from three-label to
four-label matching.
### Release note
Add the storage-compute separated Table Stream control plane. End-to-end
availability depends on the Cloud Row Binlog data plane in #65810.
### Check List (For Author)
- Test
- [x] Regression test
- `table_stream_p0` and `row_binlog_p0`: 15 suites passed.
- [x] Unit Test
- 73 FE Table Stream, transaction, drop, planner, and MV rewrite
tests passed.
- Cloud MetaService, key, clone-chain, recycler, checker, HTTP, and
metric tests passed.
- [x] Manual test
- `./build.sh --cloud` passed.
- `NODE_OPTIONS=--openssl-legacy-provider ./build.sh --fe` passed
with Checkstyle.
- Cloud C++ formatting and changed-line clang-tidy checks passed;
two pre-existing test-file analysis failures remain unrelated to this PR.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Cloud Table Stream uses MetaService-managed offsets,
immediate atomic consumption commits, and FORCE-only drop semantics. Non-Cloud
behavior is unchanged.
- Does this need documentation?
- [ ] No.
- [x] Yes. The implementation and test designs have been reviewed;
public user documentation should be published when #65810 completes the
end-to-end feature.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
