MalikHou opened a new issue, #3212:
URL: https://github.com/apache/brpc/issues/3212
# [Feature Request] Run-to-Completion (RTC) Worker Loop + User Hooks +
Optional io_uring Backend
## Background
We run brpc in a **per-thread-per-core** deployment where worker threads are
CPU-pinned and also interact with a high-performance I/O stack (SPDK or
io_uring). In this model, there are scheduling “gaps” inside the worker loop
(e.g., after running a batch of runnable bthreads and before the worker blocks
in `epoll_wait`). Those gaps are ideal to execute small, non-blocking work **on
the same worker thread** to preserve locality and avoid cross-thread handoff.
Concrete examples of work we want to run in these gaps:
- Harvest/poll **SPDK** completions (poll-mode style)
- Drain **io_uring** CQEs and advance state machines
- Convert completions into runnable units and **push them back to the
worker’s local queue**
Today, integrating these cleanly requires extra polling threads, periodic
timers, or cross-thread scheduling, which increases tail latency and adds
contention.
## Problem
1. **No first-class “run user logic on the current worker thread” hook
points**
- Users cannot reliably execute a lightweight function at well-defined
points in the worker loop (before/after waiting, between batches, etc.).
2. **No supported way to reschedule derived work back into the *local* run
queue**
- For completion-driven workloads, the critical requirement is: harvest
completions on the worker thread and immediately enqueue follow-up work to the
**same worker’s local queue**. Global queues/cross-thread wakeups increase
contention and latency.
3. **epoll-centric waiting is not ideal for io_uring/SPDK-centric stacks**
- For io_uring, batching submit/completion is the natural model and can
reduce syscalls.
- For SPDK, polling is the natural model and requires a bounded busy loop
or hybrid strategy.
## Requirements
### R1. Run-to-Completion (RTC) style worker execution
Provide an opt-in worker model that repeatedly:
- runs runnable bthreads until a budget is reached,
- runs registered “maintenance” hooks (budgeted),
- waits for events (blocking/polling/hybrid),
- repeats.
### R2. epoll polling / hybrid mode
Support configurable event-wait behavior for the epoll backend:
- **blocking** (default): `epoll_wait(timeout > 0)`
- **polling**: `epoll_wait(timeout = 0)` in a bounded busy loop
- **hybrid**: poll for `N` iterations or `T` microseconds, then fall back to
blocking
### R3. User hook registration on worker threads
Provide an API to register callbacks that execute **on the current brpc
worker thread** at defined hook points. Hooks must be:
- non-blocking (no sleeping, no long syscalls)
- budgeted (time/ops limited)
- safe to run frequently
### R4. Ability to enqueue work back to the worker’s *local* queue
Hooks must be able to take harvested completions and schedule follow-up work
back to the worker’s local run queue (not global), e.g.:
- enqueue an existing bthread/task handle locally
- or submit a closure that is executed by the same worker
### R5. Optional io_uring backend replacing epoll
Support selecting an **io_uring** event backend as an alternative to epoll
(opt-in), to:
- reduce syscalls via batching
- align with per-thread-per-core (one ring per worker)
- enable future enhancements (multishot, buffer rings) without forcing epoll
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
