bobhan1 opened a new pull request, #67989:
URL: https://github.com/apache/doris/pull/67989
### What problem does this PR solve?
A retry of an ongoing lazy commit can scan the last few temporary rowsets
and select immediate commit. If lazy publication clears the partition's pending
and temporary keys before the immediate write transaction starts, while the
global transaction remains `COMMITTED`, immediate commit reuses the stale scan
and publishes the same physical rowset at the next partition version. This can
duplicate rowsets and their statistics.
Batch-read every candidate temporary key without snapshot semantics in the
same KV transaction that publishes the rowsets, after any pending-transaction
wait. A key missing at the transaction's read version rejects the entire
attempt with `KV_TXN_CONFLICT`; a later concurrent deletion creates a real
commit conflict. The existing proxy retries the complete handler, clears the
response, and rescans. Lazy partition concurrency and the existing
state/response protocols are preserved.
The two commits separate reproduction from the fix:
1. `d12f4bf347d`: add deterministic UTs and synchronization points. Two lazy
partition tasks are in flight together; the retry scans the last rowset before
lazy removes it, then resumes while the global transaction is still `COMMITTED`.
2. `f8c2b70ab6a`: add validation and error handling only in
`commit_txn_immediately()`.
### Release note
Fix duplicate rowset publication and duplicated statistics when a lazy
commit retry switches to immediate commit.
### Check List (For Author)
- Test
- [x] Unit Test
- Before the fix: 12 of the 14 new tests fail as expected; both
pending-wait controls pass. Both MemTxnKv and real FoundationDB reproduce
versions 2 and 3 for the same physical rowset.
- After the fix: all 14 new tests pass. Coverage includes stale
scans, one missing candidate across multiple partitions, actual FDB conflicts
caused only by deleting a temporary key, pending waits, proxy
rescanning/response clearing, read errors, retry switches/limits, and normal
immediate/VISIBLE retries. Full KV snapshots verify that failed attempts leave
no persistent changes.
- Complete `txn_lazy_commit_test`: **39 passed**, with 1
pre-existing disabled test.
- Related `meta_service_test` commit, 2PC, MOW, versioned, and Table
Stream cases: **24 passed**.
- Cloud UTs built and run with ASAN. Modified lines formatted with
clang-format 16; `git diff --check` passed.
- Behavior changed:
- [x] Yes. Stale immediate-commit candidates return a retryable conflict
instead of being published again.
- Does this need documentation?
- [x] No.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
