ai-yang opened a new issue, #10936: URL: https://github.com/apache/rocketmq/issues/10936
## Before Creating the Enhancement Request - [x] I have confirmed that this should be classified as an enhancement rather than a bug or feature. ## Summary Optimize `MessageDecoder.encodeMessages(List<Message>)` so that a batch is encoded directly into one exactly sized output buffer instead of allocating a full encoded `byte[]` for every message and then copying all of those arrays into a second final buffer. The proposed implementation preserves the existing public API and produces byte-for-byte identical wire data. ## Motivation `encodeMessages` is used in the producer batch path, including automatic message batching. Its current implementation performs the following work for a batch of `N` messages: 1. call `encodeMessage` and allocate one complete encoded array for each message; 2. retain all `N` arrays in a list; 3. allocate the final aggregate array; 4. copy every encoded array into the aggregate array. That creates avoidable allocation pressure and memory bandwidth on a producer hot path. A repeated JMH 1.36 benchmark measured both throughput and normalized allocation. Its `baseline()` method reconstructs the previous aggregation algorithm in the same benchmark binary: encode each message fully, retain the arrays, then copy them into the final output. Benchmark controls: - one thread pinned to CPU 5; - 3 forks, 5 warm-up iterations and 7 measurement iterations per fork; - each iteration 500 ms; - `-Xms512m -Xmx512m`; - JMH GC profiler; - OpenJDK 11.0.31 on Ubuntu 22.04.4 / Intel Xeon Gold 6133. Each throughput score is the mean plus or minus its JMH 99.9% confidence-interval half-width. | Scenario | Reconstructed baseline ops/s | Proposed ops/s | Throughput | Baseline B/op | Proposed B/op | Allocation | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 10 messages x 128 B | 340,743 +/- 6,348 | 381,842 +/- 9,236 | +12.06% | 6,760 | 4,552 | -32.66% | | 100 messages x 128 B | 32,247 +/- 1,099 | 36,246 +/- 242 | +12.40% | 67,152 | 45,192 | -32.70% | | 10 messages x 1 KiB | 179,223 +/- 3,518 | 240,718 +/- 10,326 | +34.31% | 24,680 | 13,512 | -45.25% | | 100 messages x 1 KiB | 16,954 +/- 594 | 22,411 +/- 594 | +32.19% | 246,352 | 134,792 | -45.28% | The reconstructed baseline and proposed 99.9% throughput confidence intervals do not overlap in any of the four scenarios. The representative 100-message x 1 KiB scenario was also repeated on OpenJDK 8u492: - throughput: 14,472.346 +/- 329.526 to 18,954.609 +/- 113.257 ops/s (**+30.97%**); - allocation: 288,568 to 173,832 B/op (**-39.76%**); - the throughput 99.9% confidence intervals are disjoint. To independently verify the reconstructed baseline, I also ran the actual production `encodeMessages` implementation from separate baseline and optimized checkouts in A-B-A order for the 100-message x 1 KiB scenario: | Production checkout | Throughput ops/s | 99.9% CI | Allocation B/op | | --- | ---: | ---: | ---: | | Baseline A1 | 17,352.526 +/- 400.418 | [16,952.108, 17,752.943] | 246,352.044 | | Optimized B | 22,600.219 +/- 880.650 | [21,719.569, 23,480.870] | 134,792.035 | | Baseline A2 | 16,876.707 +/- 671.661 | [16,205.046, 17,548.368] | 246,352.045 | The optimized production checkout is 30.24% faster than A1 and 33.91% faster than A2, while allocating 45.28% less. Its 99.9% confidence interval does not overlap either baseline run. ## Describe the Solution You'd Like Use two passes without constructing temporary full-message encodings: 1. For every message, retain the body reference, serialize its properties exactly once, cache those property bytes, and calculate its encoded size. 2. Allocate one `ByteBuffer` for the exact aggregate size. 3. Write each message's total size, magic code, CRC placeholder, flag, body and properties directly into that shared buffer. The single-message `encodeMessage` method can use the same internal size and field-writing helpers, keeping the wire layout in one implementation. Add a compatibility test which asserts that batch encoding is byte-for-byte equal to concatenating the existing single-message encoding. The test should include empty and large bodies, Unicode and multiple properties, edge flag values, and an empty batch. Add a fixed golden-vector test as an independent oracle for the wire field layout, because the single-message and batch paths share the refactored helper. ## Describe Alternatives You've Considered - **Keep the current per-message arrays and final copy:** simplest, but retains the allocation and copy cost shown above. - **Use a dynamically growing output buffer:** avoids the size pass but introduces capacity growth, over-allocation, and potentially more copies. - **Calculate sizes first but serialize properties again while writing:** avoids full-message arrays but duplicates property serialization work. Caching the serialized properties keeps the two-pass design deterministic and avoids that extra CPU cost. ## Additional Context This is a focused follow-up to [#2931](https://github.com/apache/rocketmq/issues/2931) and the unmerged PR [#2932](https://github.com/apache/rocketmq/pull/2932), which had a similar copy-reduction motivation in 2021 but bundled unrelated broker, store and topic changes and later became stale. The proposed production change is limited to `MessageDecoder`, adds wire-compatibility coverage, serializes properties once, and includes multi-fork benchmark evidence. Active issue [#10442](https://github.com/apache/rocketmq/issues/10442) and PR [#10444](https://github.com/apache/rocketmq/pull/10444) optimize property encoding/decoding allocation in `MessageDecoder`, but do not change `encodeMessages` or eliminate its per-message full buffers and final copies. PR [#10588](https://github.com/apache/rocketmq/pull/10588) concerns propagation of batch message properties. Both are conceptually separate from this optimization. The full `common` module test suite passes with 245 tests and no failures, errors or skips. Three downstream store batch suites pass 8/8 tests, and the producer accumulator suite passes 3/3 tests on this performance branch. Benchmark source, methodology, checksums, and raw per-fork/per-iteration results: https://gist.github.com/ai-yang/c3e3f65351054263dec6f7b8d67ad3cd -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
