FangzuoZhang opened a new pull request, #3484: URL: https://github.com/apache/brpc/pull/3484
### What problem does this PR solve? Issue Number: N/A Related work: - https://gitcode.com/boostkit/brpc/issues/17 - https://gitcode.com/boostkit/brpc/merge_requests/32 Problem Summary: `InputMessenger` currently schedules each parsed input message in a separate bthread. For short message handlers and bursty traffic on a single connection, bthread creation and scheduling may account for a significant portion of the request-processing overhead. This PR introduces optional input message batching. Messages parsed from the same socket can be processed sequentially in one bthread, reducing scheduling overhead while preserving the existing behavior by default. ### What is changed and the side effects? ### Design ```text Parsed messages m1 m2 m3 ... mN | v +--------------------------+ | Batch Size Controller | | fixed or adaptive (EMA) | | size: 1 / 2 / 4 / 8 / 16| +------------+-------------+ | v +--------------------------+ | InputMessageBatch | | [m1, m2, ... , mk] | +------------+-------------+ | v One bthread | v m1 -> m2 -> ... -> mk processed sequentially ``` Changed: - Add the experimental `input_message_batch_process_size` gflag: - `0` or `1`: preserve the original one-message-per-bthread behavior. - Values greater than `1`: use a fixed batch size. - `-1`: adaptively select a batch size from `1`, `2`, `4`, `8`, and `16`. - Values smaller than `-1` are rejected. - Add `InputMessageBatch` to own and process messages in their original order. - Add per-socket adaptive state based on an exponentially weighted moving average of messages parsed from each read. - Increase the adaptive batch size gradually and decrease it more quickly when the observed burst size drops. - Reset adaptive history after switching away from adaptive mode. - Support batch scheduling in TCP, RDMA, and UBShm/UBRing transports. - Preserve the existing last-message scheduling optimization. - Keep progressive-read messages on the individual-message path. - Disable batching when user code runs in coroutine mode. - Fall back to synchronous processing if batch allocation or bthread creation fails. Side effects: - Performance effects: The default value is `0`, so existing deployments retain the original scheduling behavior. When batching is enabled, it reduces bthread creation and scheduling overhead for bursty workloads. A larger fixed batch may increase the time that later messages wait behind earlier handlers. Adaptive mode limits the maximum batch size to 16 and decreases the batch size quickly when the observed burst size drops. - Breaking backward compatibility: There is no change to the public RPC protocol or default runtime behavior. The internal `Transport` interface gains a `QueueMessages` virtual method. Downstream custom transport implementations derived directly from `Transport` must implement this method. ### Performance test The RDMA performance example was used with a single connection and multiple outstanding requests on that connection. Each attachment size was tested three times. - Baseline: `input_message_batch_process_size=0` - Optimized: `input_message_batch_process_size=-1` - `queue_depth` must be greater than 1 to produce message bursts on the same connection. - Latency values are in microseconds. - CPU utilization may exceed 100% because it represents multi-core process CPU usage. - QPS improvement is calculated against the average QPS of the unoptimized baseline. #### Average results | Attachment | Avg Latency | P90 | P99 | Baseline QPS | Batched QPS Avg | Server CPU | Client CPU | QPS Improvement | |---|---:|---:|---:|---:|---:|---:|---:|---:| | 0 bytes | 351.00 | 543.00 | 810.00 | 2684.100 | 2886.265 | 1336.00% | 3950.67% | +7.53% | | 256 bytes | 398.00 | 610.67 | 927.00 | 2377.408 | 2545.978 | 1315.00% | 4798.00% | +7.09% | | 1 KB | 438.33 | 683.67 | 1044.67 | 2174.254 | 2320.362 | 1322.67% | 3732.67% | +6.72% | | 4 KB | 660.67 | 1135.67 | 2123.33 | 1486.802 | 1544.824 | 1270.00% | 2577.33% | +3.90% | | 8 KB | 1031.00 | 2310.00 | 5289.33 | 1049.357 | 1104.763 | 1177.00% | 1993.33% | +5.28% | | 100 KB | 7388.67 | 12362.00 | 22406.67 | 134.602 | 138.385 | 925.33% | 988.00% | +2.81% | --- ### Check List: - Please make sure your changes are compilable. - When providing us with a new feature, it is best to add related tests. - Please follow [Contributor Covenant Code of Conduct](https://github.com/apache/brpc/blob/master/CODE_OF_CONDUCT.md). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
