This is an automated email from the ASF dual-hosted git repository. RongtongJin pushed a commit to branch codex/dledger-latest-pr336-adapter in repository https://gitbox.apache.org/repos/asf/rocketmq.git
commit 85872858ec38a3a1f99a7e2d61f7f39406136818 Author: 通融 <[email protected]> AuthorDate: Fri Aug 14 16:04:09 2026 +0800 docs: design latest DLedger PR336 adaptation --- ...-08-14-dledger-latest-pr336-fastjson2-design.md | 116 +++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/docs/superpowers/specs/2026-08-14-dledger-latest-pr336-fastjson2-design.md b/docs/superpowers/specs/2026-08-14-dledger-latest-pr336-fastjson2-design.md new file mode 100644 index 0000000000..d08b95a33a --- /dev/null +++ b/docs/superpowers/specs/2026-08-14-dledger-latest-pr336-fastjson2-design.md @@ -0,0 +1,116 @@ +# Latest DLedger PR #336 RocketMQ Fastjson2 Migration Design + +**Date:** 2026-08-14 + +## Context + +RocketMQ still consumes DLedger 0.3.2. Pull request apache/rocketmq#10650 upgrades that dependency to DLedger 0.3.3-SNAPSHOT and adapts RocketMQ to the current DLedger API, but it is based on an old RocketMQ commit and leaves Maven/Bazel dependency and compatibility-test gaps. + +DLedger pull request openmessaging/dledger#336 fixes a regression in current DLedger master: a locally successful append returns an `AppendFuture` whose position is still `-1`. RocketMQ reads that position before quorum completion in both the commit log and controller paths, so the regression makes a valid append look like a failure. + +The existing 0.3.2-maintenance solution remains unchanged on its own branches. This design creates an independent latest-code validation path. + +## Source Baselines + +- RocketMQ: `develop` at `293f5885719fc4aa3619446a1900f58ccfcfdd29`. +- DLedger: current `master` at `2834424cec15670c1b1108378b0b0a3a5fd57791` plus PR #336 head `a2555fa78fcda379279a815357717283d0f6c56b`. +- fastjson2: `2.0.64` for Maven and Bazel, matching apache/rocketmq#10928 and avoiding the JDK 8 initialization-order regression in 2.0.61 through 2.0.63. + +The DLedger validation artifact will use a unique local version such as `0.3.3-pr336-f2-64-SNAPSHOT`. Version and fastjson2 metadata changes are build-only overlays; PR #336 remains a focused append-position fix. + +## Goals + +1. Port the useful DLedger API adaptation from RocketMQ PR #10650 onto current RocketMQ develop. +2. Consume current DLedger plus PR #336 and prove the append position is available immediately for single and batch appends. +3. Remove fastjson1 from RocketMQ and DLedger resolved dependencies, test classpaths, Bazel artifacts, and packaged bytecode. +4. Preserve the ability to read representative JSON produced by fastjson 1.2.83 without executing fastjson1 in tests. +5. Validate RocketMQ behavior across storage, controller, broker lifecycle, production/consumption, failover, and restart paths. + +## Non-Goals + +- Mixed-version rolling operation between DLedger 0.3.2 and current master. +- Direct downgrade from current DLedger data to 0.3.2. +- Push-protocol compatibility bridging between the old `entry`/`batchEntry` schema and the new `entries`/`preLogIndex`/`preLogTerm` schema. +- Unrelated RocketMQ or DLedger refactoring. + +The result may support a coordinated full-stop upgrade after validation, but it must not claim rolling-upgrade compatibility. + +## RocketMQ Changes + +### DLedger API adaptation + +Port PR #10650 by intent rather than cherry-picking its stale commit: + +- Import `AppendFuture` and `BatchAppendFuture` from `io.openmessaging.storage.dledger.common`. +- Replace `setEnableBatchPush` with `setEnableBatchAppend`. +- Read the committed index through `MemberState` where the latest store API removed it. +- Adapt the controller state machine from `CommittedEntryIterator` to `ApplyEntryIterator` and unwrap `ApplyEntry`. +- Derive a committed physical position from the committed entry because `getCommittedPos()` no longer exists. +- Update Maven and Bazel DLedger coordinates together. + +No PR #10650 code is accepted blindly. Each change is checked against the exact PR #336 artifact and current RocketMQ source. + +### Complete fastjson1 removal + +- Remove `com.alibaba:fastjson` from root Maven dependency management and `common/pom.xml`. +- Remove the fastjson1 artifact from `WORKSPACE` and its test dependency from `remoting/BUILD.bazel`. +- Pin fastjson2 2.0.64 in both Maven and Bazel. +- Preserve the DLedger POM exclusion for `com.alibaba:fastjson`; the exclusion is allowed as guard metadata and must not resolve an artifact. +- Replace the live fastjson1 compatibility oracle with checked-in JSON literals generated by fastjson 1.2.83. + +Golden fixtures cover `BatchAck`, `SubscriptionData`, and `ConsumerConnection`, including byte/bit encoding, nested collections, enums, renamed fields, and fields marked `serialize = false`. + +## Data Flow and Correctness + +For a RocketMQ write, `DLedgerServer` appends locally before quorum acknowledgement. PR #336 sets the returned future position to the appended entry position before returning. RocketMQ then computes the message physical offset without mistaking the append for `OS_PAGE_CACHE_BUSY`. Batch append uses the last entry as the base future position and retains every individual position in `BatchAppendFuture#getPositions()`. + +Controller append follows the same immediate-position contract. State-machine application uses the latest `ApplyEntryIterator` API and applies the embedded DLedger entry body exactly once. + +Committed-position derivation receives additional restart coverage. Current DLedger initializes `MemberState.committedIndex` to `-1` and no longer restores the old checkpoint value. The validation must therefore cover an existing-data full restart before any new user write. If that test exposes an incorrect max offset, read result, or role transition, a minimal RocketMQ fix will be designed from the failing test without treating an uncommitted tail as committed. + +## TDD and Verification Strategy + +### DLedger gates + +- Reproduce the current-master failure where single and batch futures return `-1` immediately after local append. +- Verify PR #336 makes the same assertions pass before quorum completion. +- Run the DLedger unit reactor and package the unique local artifact with JDK 8 and fastjson2 2.0.64. + +### RocketMQ compile and dependency gates + +- First compile current RocketMQ against the PR #336 artifact to capture the expected latest-API failures. +- Port the minimum API changes until Maven and Bazel compile. +- Verify Maven dependency trees resolve one DLedger artifact and fastjson2 2.0.64, with no fastjson1. +- Verify Bazel resolves the same versions and excludes DLedger's external RocketMQ remoting artifact in favor of the reactor target. + +### Focused behavior tests + +- `RemotingSerializableCompatTest`, including the three fastjson1 golden fixtures and fresh-JVM JDK 8 cold start. +- DLedger controller append and state-machine application tests. +- DLedger commit-log single and batch append tests, including immediate future positions. +- Broker container add/remove DLedger broker lifecycle. +- Connection-failure callbacks for DLedger RPC futures. + +### Integration and recovery gates + +- Real DLedger produce-and-consume integration test. +- Three-node single and batch append, leader election, leader shutdown, follower promotion, and continued production/consumption. +- Controller leader change and committed-index convergence. +- Existing data full-cluster shutdown and restart. Before a new write, verify max offset, message reads, dispatch progress, and broker role transition; then append and consume new messages and restart once more. +- Run relevant Maven packaging/tests and Bazel builds/tests on JDK 8. + +Tests that create DLedger stores use the low-usage isolated APFS volume. A `DISK_FULL` result on the host data volume is an environment failure and is rerun only after the test JVM is proven to use the isolated volume. + +## Acceptance Criteria + +- All targeted Maven and Bazel builds and tests pass. +- DLedger single and batch futures expose correct positions before quorum completion. +- RocketMQ production, consumption, controller, failover, and existing-data restart scenarios pass. +- `mvn dependency:tree`, Bazel resolution output, source scans, JAR entry scans, and class constant-pool scans contain no resolved or embedded fastjson1. +- fastjson1 1.2.83 golden JSON remains readable through production `RemotingSerializable` APIs. +- Maven and Bazel both use fastjson2 2.0.64 and the same unique DLedger validation coordinate. +- The existing maintenance-line solution and its branches remain unchanged. + +## Delivery Boundary + +The first result is a local validation branch and evidence report. It is not pushed as a RocketMQ community PR while it depends on a local-only DLedger coordinate. After PR #336 is merged and a consumable DLedger version exists, the coordinate is replaced with the public version and the clean Maven/Bazel/integration matrix is rerun before publication.
