lizining1231 opened a new pull request, #3718:
URL: https://github.com/apache/dubbo-go/pull/3718
### Description
Fixes #3717
Triple unary RPCs serialize every payload through `codec.Marshal`, which
calls `proto.Marshal`. Because `proto.Marshal` always starts from `nil`, each
message pays a structural per-request allocation tax: a fresh `[]byte` of ~L
bytes is allocated and zeroed (`memclr`) from scratch, then
`bytes.NewBuffer(raw)` wraps it with a ~40B escaping allocation before the
bytes are written to the transport.
1. **Per-message output slice (optimization point A)**: `proto.Marshal`
builds the encoded payload from a `nil` starting point, so the slice can never
reuse the pooled capacity that `bufferPool` already holds. Each message
allocates ~L bytes and memclr-zeroes them (~53µs at 1MiB), even though the pool
hands out a reusable `*bytes.Buffer` on the same call path.
2. **Fresh wrapper + transport copy (optimization point B)**: the freshly
allocated slice is wrapped by `bytes.NewBuffer(raw)` (a ~40B escaping
allocation) and then copied wholesale into the transport writer, while the
pooled buffer underneath is returned without its underlying array ever being
reused.
3. **Server-side generic responses allocate twice**: non-IDL server
responses are wrapped in a `TripleResponseWrapper` by
`tripleServerCodecSession`, serializing the payload and the wrapper as two
separate allocations.
The fix mirrors the approach upstream connect-go already ships
(`marshalAppend`): marshal into the caller-provided pooled buffer instead of
into a fresh slice. It is purely additive, codecs without the extension are
untouched and keep their exact previous behavior.
### Changes
**1. Optional `marshalAppender` codec extension (non-breaking)**
`protocol/triple/triple_protocol/codec.go`:
- Add optional interface `marshalAppender { MarshalAppend(dst []byte,
message any) ([]byte, error) }`; implementing codecs serialize into a
caller-provided buffer. The `Codec` interface itself is unchanged, so
hessian2/json/msgpack and third-party codecs compile and behave exactly as
before.
- `protoBinaryCodec.MarshalAppend` delegates to
`proto.MarshalOptions{}.MarshalAppend`, which drives the same protobuf encoder,
so output is byte-identical to `Marshal`, and the cap-gated pure-append path
means zero allocation when the buffer's spare capacity suffices.
- `tripleServerCodecSession.MarshalAppend` extends the per-request server
codec: the IDL leg forwards to the delegate's appender, and the non-IDL leg
marshals the inner payload with its own codec but appends the outer
`TripleResponseWrapper` into the caller-provided buffer instead of allocating a
fresh slice, so server responses reach the fast path too.
`protocol/triple/triple_protocol/buffer_pool.go`:
- Add shared helper `marshalToPool(pool, appender, message)`: borrow a
`*bytes.Buffer` from `bufferPool`, call `MarshalAppend` into its spare
capacity; when the appender had to grow the slice, swap the larger array back
into the pooled buffer so the capacity is recycled instead of dropped; on
failure the buffer is returned and a `CodeInternal` error surfaces. Both
marshalers now share one borrow/marshal/adopt dance.
**2. Fast path in both unary marshalers + a shared compression tail**
`protocol/triple/triple_protocol/envelope.go` (`envelopeWriter.Marshal`,
gRPC/Triple envelope wire):
- Probe the codec for `marshalAppender`; on hit run the new
`marshalAndWrite` (marshalToPool → `Write(&envelope{Data: buffer})`),
preserving the backup-codec fallback semantics of the slow path. On miss the
existing `marshalWithFallback` slow path is kept unchanged.
`protocol/triple/triple_protocol/protocol_triple.go`
(`tripleUnaryMarshaler.Marshal`, Triple unary body wire):
- Same probe: capable codecs take the new `marshalAndWrite` fast path;
codecs without the extension keep the old `codec.Marshal` slow path unchanged.
- The compression tail previously inlined at the end of `Marshal` is
extracted into one shared `compressAndWrite`, called by both the fast and slow
paths, so the compression threshold, pooled gzip, `sendMaxBytes` enforcement,
compression header and final write can never drift between the two.
```mermaid
flowchart TD
classDef base fill:#f6f8fa,stroke:#8b949e
classDef added fill:#fff3b0,stroke:#e3a008,stroke-width:2px
C["Client sends request<br/>conn.Send → marshaler<br/>codec = configured
codec (proto → protoBinaryCodec)"]:::base
S["Server sends response<br/>conn.Send → marshaler<br/>codec = wrapped
as tripleServerCodecSession (supports generic calls)"]:::base
M["tripleUnaryMarshaler.Marshal (L504)<br/>Shared by both sides"]:::base
P["Capability probe: m.codec.(marshalAppender)"]:::added
F["Optimized path<br/>marshalAndWrite → marshalToPool<br/>→
MarshalAppender with buffer pooling → zero allocation<br/>(Implementers:
protoBinaryCodec / server session wrapper)"]:::added
SL["Legacy path<br/>codec.Marshal → always allocates"]:::base
T["compressAndWrite (L540)<br/>Compression / size limit / write header /
write body<br/>Common tail path shared by fast & slow (extracted in this
change)"]:::added
O["HTTP body"]:::base
C --> M
S --> M
M --> P
P -->|hit| F
P -->|miss| SL
F --> T
SL --> T
T --> O
```
### Test
`marshal_perf_regression_test.go`
| Test | Description |
|---|---|
| `TestMarshalPerfWireParity` | Core invariant: for the same message and
compression setting the MarshalAppend fast path emits byte-for-byte the same
wire output as the slow `codec.Marshal` path, on both the triple body wire and
the envelope wire |
| `TestMarshalPerfPoolInvariants` | `bufferPool.Get` returns an empty
buffer, `Put` resets length; `nil` message writes nothing; an empty proto
yields a zero-length (5-byte prefix only) envelope; the `compressMinBytes`
boundary anchored on the real encoded length never panics and both paths agree
on the compression flag |
| `TestMarshalPerfBackupCodecFallback` | A failing codec implementing both
`Codec` and `marshalAppender` forces both paths into the backup-codec fallback;
output is byte-identical to the healthy reference, with no double fallback,
`CodeInternal` on final error, and no panic when the backup is nil |
| `TestMarshalPerfCompressionAndMaxBytes` | Both paths enforce the
compression threshold and `sendMaxBytes` identically: over-limit messages
return `CodeResourceExhausted`, compressed messages set the `Content-Encoding`
header on both |
| `TestMarshalPerfLargeBufferDropped` | Buffers grown beyond the 8MiB
recycle cap are dropped, not reused, and a large message leaves no residue for
the next small message |
| `TestMarshalPerfConcurrentSend` | Concurrent sends share the same
marshaler and `bufferPool` with no data races (run under `-race`) |
| `TestMarshalPerfTypeGuard` | Scope guard: codecs whose `MarshalAppend` is
not byte-identical to `Marshal` (hessian2/json/msgpack/backup wrapper) must not
implement `marshalAppender`; only `protoBinaryCodec` and
`tripleServerCodecSession` do |
| `TestMarshalPerfErrorGuard` | A non-proto message never panics on either
path and surfaces as `CodeInternal`, matching slow-path behavior |
| `TestMarshalPerfServerSessionFastPath` | `tripleServerCodecSession`
reaches the MarshalAppend branch end to end for both IDL and non-IDL responses
with byte-identical output |
`marshal_perf_bench_test.go`
| Benchmark | Description |
|---|---|
| `BenchmarkUnaryMarshalerFastPath` | `tripleUnaryMarshaler.Marshal` with
`protoBinaryCodec` (MarshalAppend fast path), payloads 128B / 1KiB / 16KiB /
1MiB |
| `BenchmarkUnaryMarshalerSlowPath` | Control: same marshaler with
`noAppenderCodec` (no `marshalAppender`, old `codec.Marshal` path), same
payloads |
| `BenchmarkUnaryMarshalerFastPathCompressed` / `SlowPathCompressed` |
Fast/slow A/B on the gzip-compressed branch (second pooled buffer, compression
header, compressed-size limit check) |
| `BenchmarkEnvelopeWriterFastPath` / `SlowPath` | Same A/B for
`envelopeWriter.Marshal` (gRPC/Triple envelope wire) |
| `BenchmarkEnvelopeWriterFastPathCompressed` / `SlowPathCompressed` | Same
A/B on the compressed envelope branch |
### Validation
#### pprof CPU Profile diff
<img width="1915" height="958" alt="image"
src="https://github.com/user-attachments/assets/da23768b-0a92-4a41-93ab-0634ac295604";
/>
#### Marshal Fast-Path Benchmark Results
**go test bench -bench=BenchmarkMarshal**
**allocs/op reduced from 3 to 1**
| Message | Before | After |
| ------- | ------------------ | ----------------- |
| 128B | 3 | 1 |
| 1KiB | 3 | 1 |
| 16KiB | 3 | 1 |
| 1MiB | 3 | 1 |
Both wire slow paths incurred 3 allocations per op, coming from proto
marshal + gzip compression buffer + envelope/header overhead. After
optimization, only 1 allocation remains, from the gzip compression pool's
initial buffer, which is then reused by the pool.
**B/op reduced by approximately −99.8%**
| Message | Before | After | Reduction |
| ------- | ------------- | ------------ | --------- |
| 128B | 237 | 6 | -97.47% |
| 1KiB | 1252 | 7 | -99.44% |
| 16KiB | 18712 | 23 | -99.88% |
| 1MiB | 1069533 | 3482 | -99.67% |
Before optimization, B/op ≈ message size, with full payload copy/allocation
per request. After optimization, only fixed overhead remains, and it no longer
grows with payload size.
Reproduce:
```bash
# pprof
go test -run '^$' -bench '^BenchmarkUnaryMarshalerSlowPath$'
-benchtime=30000x \
-cpuprofile cpu_slow.out ./protocol/triple/triple_protocol/
go test -run '^$' -bench '^BenchmarkUnaryMarshalerFastPath$'
-benchtime=30000x \
-cpuprofile cpu_fast.out ./protocol/triple/triple_protocol/
go tool pprof -top -diff_base cpu_slow.out cpu_fast.out
# go test bench
go test -run '^$' -bench
'^BenchmarkEnvelopeWriter(FastPath|SlowPath)Compressed$' \
-benchmem -count=3 ./protocol/triple/triple_protocol/
```
### Checklist
- [x] I confirm the target branch is `develop`
- [x] I have run `make fmt` to format my code
- [x] I have run `make test` to run local tests
- [x] I have added tests that prove my fix is effective or that my feature
works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
