qianye1001 opened a new pull request, #11490:
URL: https://github.com/apache/rocketmq/pull/11490
### Which Issue(s) This PR Fixes
- Fixes #11489
### Brief Description
With POP KV enabled, storage ACKs and invisible-time changes currently
execute synchronous RocksDB writes on the shared ACK executor. Bursts of CHANGE
requests can occupy its workers waiting for WAL synchronization or write
stalls, increasing ACK latency and reducing consumption throughput.
This change queues POP KV mutations for a single group-commit writer. It
drains available requests into an ordered `WriteBatch` without waiting to
collect a batch, retains `sync=true` and WAL, and completes persistence Futures
on a separate completion thread. ACK, batch ACK and CHANGE responses now wait
asynchronously for the required writes; the POP KV CHANGE path no longer blocks
the request executor when the legacy async flags are disabled.
The same writer handles POP checkpoints and background maintenance. With the
cache disabled, CHANGE replaces the old checkpoint atomically; with the cache
enabled, the replacement becomes durable before removing the old cached
checkpoint. Same-key replacements avoid deleting the new record, and revive
advances its scan cursor only after persistence succeeds. Failed POP checkpoint
writes now return an error response and release message buffers.
Admission is bounded to 4096 outstanding requests and 64 MiB of encoded
mutations. Batches contain up to 128 requests with a 1 MiB target; a larger
atomic request is kept intact. Full/stopped admission fails immediately,
accepted requests drain before RocksDB closes, and synchronous background
callers also use the shared writer.
No client or wire-protocol changes are required. This complements the batch
CHANGE protocol proposed in #10489.
### How Did You Test This Change?
Using JDK 21 after initializing the upstream protocol submodule:
```shell
git submodule update --init rocketmq-apis
mvn -B -pl broker -am test \
-Dtest='PopConsumer*Test,PopKVAsyncProcessorTest,AckMessageProcessorTest,ChangeInvisibleTimeProcessorTest,PopMessageProcessorTest'
\
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
-Djacoco.skip=true
```
Result on the rebased branch: **83 tests, 0 failures, 0 errors, 1
pre-existing skipped test**. Checkstyle and SpotBugs passed.
Validation includes real RocksDB reopen/snapshot/replacement checks, ordered
batching, write-failure propagation, queue and byte limits, shutdown draining,
callback isolation, cached-checkpoint preservation, same-key changes,
scan-cursor protection, and response timing for ACK/batch ACK/CHANGE. An
EmbeddedChannel regression verifies that POP persistence rejection produces an
error response.
JaCoCo 0.8.5 is skipped because it does not support JDK 21. No end-to-end
Broker/Proxy performance or crash-recovery improvement is claimed by this PR;
those require separate workload validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
