sollhui opened a new pull request, #68385:
URL: https://github.com/apache/doris/pull/68385
### What problem does this PR solve?
Problem Summary:
Load-time bitmap computation, write-end bitmap reconciliation, cloud commit
bitmap work, and memtable flush currently use separate execution pools. A
bitmap backlog therefore does not directly reduce the rate at which flush
workers produce new segments. A global task-priority queue would also
disadvantage duplicate-key loads, which have only flush work.
Share foreground bitmap and flush workers within each existing resource
domain. Schedule one task per transaction FIFO turn, selecting P0
commit/publish, P1 write-end reconciliation, P2 write-time bitmap, then P3
flush inside that transaction. Requeue the transaction before executing the
task. There is no per-load concurrency cap or ready-queue membership flag; the
empty-to-nonempty transition and dequeue/requeue operation are serialized by
the existing pool mutex.
Retain writer/tablet token wait and cancellation boundaries, and keep
background bitmap callers on their original executor. Remove the separate
load-bitmap, cloud tablet-bitmap, and high-priority flush pools. Cloud commit
restores the workload group from transaction-cache metadata. Bitmap children
created inside a load worker execute inline to preserve tablet lock scopes
without waiting for queued work in the same pool. Cancellation removes only the
target token's queued work and does not report discarded bitmap work as
successful.
Design:
[docs/design/load-fifo-scheduling.md](docs/design/load-fifo-scheduling.md).
### Scope and tradeoffs
- Fairness is by task dispatch, without preemption or equal-CPU-time
guarantees. Inline cloud commit children reduce segment parallelism within a
tablet; different tablets can still run concurrently.
- Ordinary loads use the same `txn_id` across writing and commit. Explicit
multi-statement transactions retain subtransaction IDs while writing and use
the parent ID at commit. This patch does not add a parent/subtransaction
scheduling mapping to the write protocol.
- A tablet commit restores its first available cached workload group,
including subtransaction entries. It does not split a commit across several
groups; missing or dropped owners use the default domain.
- Existing memtable admission and memory controls provide upstream pressure.
No new bitmap-byte budget or dependency graph is added.
### Release note
Foreground load bitmap work shares memtable flush workers and receives stage
priority within transaction FIFO scheduling. Duplicate-key loads retain FIFO
dispatch turns. The former `calc_delete_bitmap_for_load_max_thread`,
`calc_tablet_delete_bitmap_task_max_thread`, and
`high_priority_flush_thread_num_per_store` settings remain parseable but no
longer size independent pools; `is_high_priority` no longer selects a separate
flush pool. Flush sizing/adaptive controls now govern shared foreground work;
`calc_delete_bitmap_max_thread` continues to size background bitmap work.
### Check List (For Author)
- Test:
- Unit tests added for queue fairness/priority, cancellation,
reactivation, unrestricted per-load parallelism, nested execution, and cleanup;
**not executed**.
- Compilation and test execution explicitly skipped at the requester's
direction. Earlier attempts stopped at dependency acquisition before C++
compilation or test execution. This is a draft awaiting validation.
- Static source review, clang-format 16 formatting, and whitespace
review performed. No throughput or latency improvement is claimed without
measurement.
- Behavior changed: **Yes**, foreground scheduling and thread-pool
allocation change as described above.
- Does this need documentation: **Yes**, design and limitations are included
in this PR.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note and retired pool configuration behavior
- [ ] Build BE and run the added unit tests
- [ ] Validate concurrent MOW/DUP, partial updates, row binlog, retries,
cancellation, and workload-group deletion
- [ ] Confirm document and scheduling tradeoffs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
