dodjdnh opened a new pull request, #3784:
URL: https://github.com/apache/dubbo-go/pull/3784

   # feat(metrics): add request-scoped RPC network and codec failure metrics
   
   ## Summary
   
   Complete the path from confirmed local Network/Codec failure origins to 
request-scoped RPC metrics, including asynchronous Dubbo response correlation 
and callback monitoring completion. Keep monitoring separate from business 
completion rather than redesigning the callback, Future, or retry contract.
   
   Related to #3774. This PR implements the bounded attribution and monitoring 
scope below; it does not claim to resolve every business-completion concern in 
that issue.
   
   ## Motivation
   
   The existing classifier relies on Triple/gRPC status codes, which cannot 
reliably distinguish transport faults from serialization failures. Both can 
surface as `Internal`, while deserialization failures can surface as 
`InvalidArgument`. Dubbo response decoding also happens on an I/O path that 
does not necessarily return its original error to the initiating Metrics Filter.
   
   Adding classification branches and counters alone is insufficient: evidence 
must belong to the correct request and sending Session, and callback submission 
must not be mistaken for completed RPC success. Fast responses, synchronous 
send errors, expiry, duplicate responses, and inherited contexts also require 
explicit monitoring ownership and completion rules.
   
   ## Changes
   
   ### Preserve confirmed local failure origins
   
   - Introduce protocol-independent Network/Codec markers that preserve error 
text and cause chains, including wrapped/joined errors and typed-nil handling.
   - Preserve Codec origins at Dubbo serializer and Triple Marshal/Unmarshal, 
Envelope, and unary boundaries. A successful fallback does not become a failed 
request.
   - Recognize network evidence only at trusted Getty write and Triple HTTP 
call boundaries with a `net.OpError`; exclude context cancellation and deadline 
errors.
   - Do not infer origins from generic status codes, error strings, or Session 
closure alone. Conflicting evidence remains Unknown.
   
   ### Correlate asynchronous evidence to its original request
   
   - Retain RequestID after a complete, validated, non-event response frame 
fails decoding, and require both its pending request and original sending 
Session to match.
   - Record evidence without removing the pending request, closing Done, 
replacing its business error, or invoking the user callback.
   - Atomically claim/remove pending responses to prevent duplicate delivery, 
and protect concurrent response-pointer access.
   - Skip decoding into a retired caller's Reply when the pending request no 
longer exists.
   
   ### Separate monitoring completion from business completion
   
   - Create one Observation per local invocation attempt and snapshot 
role/labels before sending; completion events do not retain and later read 
mutable invocation/result objects.
   - Bind Consumer monitoring to the exact invocation pointer and allow one 
remoting claim. Provider observations cannot be claimed by outgoing requests; 
inheriting a context does not grant monitoring ownership.
   - Install callback monitoring before sending. Buffer a fast response until 
Invoke returns; a synchronous send error takes precedence.
   - Resolve response, cancellation, and monitoring timeout into one monitoring 
terminal state, then release invocation/Session references, buffered state, 
observer closure, timer, and context hook.
   - Expiry without a known business result releases processing occupancy only. 
It does not invent completed totals, success/failure counts, or RT samples, 
remove the original business pending request, or suppress a late user callback.
   
   ### Classify and update metrics centrally
   
   - Count each confirmed failed attempt in total failures and exactly one 
category; Network/Codec failures do not also increment Unknown. Successful 
results remain successful.
   - Preserve existing DeadlineExceeded, ResourceExhausted, PermissionDenied, 
and BizError precedence. Unmarked Unavailable remains ServiceUnavailable.
   - Register the following metrics for both `provider` and `consumer`, using 
existing RPC labels without RequestID, Session, or error-text labels. 
Registering both sides does not imply complete Provider wire-failure coverage 
outside the Filter.
   
   | Metric | Type | Meaning |
   | --- | --- | --- |
   | `dubbo_{side}_requests_failed_network_total` | Counter | Confirmed 
Network-classified failures |
   | `dubbo_{side}_requests_failed_codec_total` | Counter | Confirmed 
Codec-classified failures |
   | `dubbo_{side}_requests_failed_network_total_aggregate` | Gauge | Network 
failures in the existing local window |
   | `dubbo_{side}_requests_failed_codec_total_aggregate` | Gauge | Codec 
failures in the existing local window |
   
   ### Add repeatable verification and document boundaries
   
   Default cross-component tests use a controlled transport with production 
Filter, DubboInvoker, ExchangeClient, event bus, Collector, and Prometheus 
registry. An opt-in loopback fixture additionally exercises real 
Getty/Dubbo/Hessian traffic and Triple HTTP dial failure.
   
   The six commits separate failure-origin preservation, metric integration, 
pending/late-response safety, decode-failure correlation, callback observation 
completion, and cross-component regression tests/documentation.
   
   ## Verification
   
   These are previously completed local test records, not a claim that upstream 
CI has passed. The callback ownership fix was tested before the final 
formatting-only history cleanup. Current HEAD is `182f2324`; its subsequent 
changes are import formatting and a test-local spelling correction, with 
lightweight checks repeated.
   
   | Area | Check | Result |
   | --- | --- | --- |
   | Unit / concurrency | Origin markers, request/Session correlation, 
observation ownership, completion races, event snapshots, Collector; core 
packages with `-race -count=3` | PASS |
   | Protocol regression | Relevant Getty, Dubbo, and Triple ordinary tests on 
Linux/WSL | PASS |
   | Static | Targeted `go vet` before formatting cleanup; gofmt, repository 
imports formatter, targeted misspell, and commit whitespace checks after 
cleanup | PASS within this scope; full lint is NOT VERIFIED |
   | Cross-component regression | Nested calls, disabled child Metrics, 
early/delayed callbacks, cancellation followed by late callback, duplicate 
responses | PASS |
   | Runtime | Opt-in real-network loopback fixture, three iterations: success, 
request encoding failure, response decoding failure, real HTTP dial failure, 
exported metrics | PASS |
   | Kubernetes E2E | RKE2 manual run covering 26 deduplicated scenarios on 
`e9c29ad7` | PARTIAL PASS; details below |
   | Supplemental verification | Five targeted Go test groups: Codec fallback, 
atomic pending claim, RequestID correlation, callback completion, Collector 
classification | PASS, 5/5; not cross-Pod E2E |
   | Commit-level regression | Callback ownership commit independently 
extracted and core race tests executed before formatting cleanup | PASS |
   | Runtime race | Unrestricted real Getty loopback fixture | FAIL: Getty 
1.4.10 teardown races also reproduced on the unmodified baseline |
   | Full lint / modernize | Complete configured analysis | NOT VERIFIED: local 
toolchain mismatch and bounded analysis timeouts during preflight |
   | Full project / upstream CI | Full build/test/race, samples integration, 
RISC-V, CodeQL, CI coverage upload | NOT VERIFIED locally |
   | Current HEAD Kubernetes rerun | Repeat the same cluster scenarios on 
`182f2324` | NOT VERIFIED |
   
   ### Kubernetes E2E: real RPC calls and metric export
   
   The existing acceptance report records a manual RKE2 run on 2026-10-28 using 
`e9c29ad7`, before the later callback ownership fix. It checks actual 
Dubbo/Triple results, callback behavior, and `/metrics`, not just metric 
registration. It is not proof of an equivalent cluster run on current HEAD.
   
   | Module | Observed behavior | Result |
   | --- | --- | --- |
   | Basic RPC | Eight scenarios: Dubbo/Triple success, business errors, 
timeouts, and Triple Unavailable/Unknown; existing categories and request 
metrics behaved as expected | PASS |
   | Network | Triple connection refusal/RST and Dubbo RST after request 
arrival; Triple Network increased by 2; Dubbo remained Unknown without reliable 
source evidence | PASS |
   | Codec | Three Dubbo cases behaved as expected: encoding failure, decoding 
failure with trusted ID, and damaged response with untrusted ID. Triple damaged 
request and response decompression failure each produced Unknown +1, Codec +0 | 
PARTIAL PASS |
   | PendingResponse | Shared-Session concurrency, late responses, cancellation 
races; four normal requests succeeded and four abnormal requests failed, but 
the latter produced Codec 1 / Unknown 3 rather than the originally expected 
Codec 4 | PARTIAL PASS |
   | Callback | Seven scenarios: success, business failure, send failure, fast 
response, duplicate response, no response, and late response; callback counts 
and final metrics matched expectations | PASS |
   | Metrics export | Real `/metrics`, Prometheus Counter values, mutually 
exclusive categories, and once-only callback accounting; Consumer Network/Codec 
metrics were exported | PASS |
   
   ### E2E outcomes that did not meet the original expectation
   
   The following explanations come from the existing fault-injection records, 
Getty logs, source inspection, and Prometheus observations.
   
   | Scenario | Actual result | Root cause / boundary |
   | --- | --- | --- |
   | Triple response decompression failure | `gzip: invalid header`; Unknown 
+1, Codec +0 | Failure occurs during local gzip decompression before Protobuf 
deserialization. Compression/decompression origin marking is outside this 
change; this is not a lost existing Codec marker. |
   | Triple damaged request | Remote InvalidArgument; Unknown +1, Codec +0 | 
Provider decoding fails before the Metrics Filter. The Consumer receives only a 
remote protocol status, not the original local Codec evidence. Arbitrary remote 
InvalidArgument must not be inferred as Codec. |
   | Dubbo concurrent damaged responses | Four failures: Codec 1, Unknown 3 | 
Getty encounters EOF decoding the first damaged response, exits its read loop, 
and closes the Session. Only that response yields trustworthy Codec evidence; 
the other requests eventually time out without individual decode evidence. The 
first response's evidence must not be assigned to all requests on the 
connection. The original Codec-4 E2E target remains unmet. |
   
   These observations have not directly demonstrated a defect in the 
request-correlation/classification implementation, but must not all be labeled 
E2E PASS. The five supplemental test groups passed and verify individual 
mechanisms; they do not replace cross-Pod reruns of unmet scenarios. Provider 
Filter-external faults and idle-window refresh are not claimed as passing E2E 
coverage.
   
   ### Local reproduction
   
   Run from the repository root on Linux/WSL with a race-capable toolchain:
   
   ```sh
   go test -race ./common/rpcerror ./filter/metrics ./metrics/rpc/... 
./remoting -count=3
   go test -p 1 ./remoting/getty ./protocol/dubbo/... 
./protocol/triple/triple_protocol -count=1
   go test -tags=rpc_failure_integration ./metrics/rpc/integration -count=3 
-timeout=60s
   go vet ./common/rpcerror ./filter/metrics ./metrics/rpc/... ./remoting 
./remoting/getty ./protocol/dubbo/... ./protocol/triple/triple_protocol
   git diff --check 97666ec89ab0c4a0fe949778f0f9a52d3b9b4f1b HEAD
   ```
   
   These commands reproduce local automation, not Kubernetes deployment/manual 
E2E. The loopback fixture is not a registry-backed deployment, and ordinary 
fixture success does not establish a clean unrestricted Getty race run.
   
   ## Review Focus
   
   1. **Architecture and responsibility:** failure origins, request 
correlation, monitoring lifecycle, and aggregation are separate layers; 
lower-level I/O does not increment RPC counters or decide business completion.
   2. **Identity and memory model:** one Observation belongs to a local 
attempt, invocation identity constrains ownership transfer, and RequestID plus 
the sending Session constrain asynchronous evidence. Please assess this 
explicit ownership/snapshot model rather than implicit sharing through 
inherited contexts.
   3. **Timing and terminal-state arbitration:** fast callbacks wait for 
Invoke's synchronous outcome, and expiry is not business completion. Please 
assess synchronous-error precedence and the decision not to fabricate a result 
on expiry.
   4. **Evidence and compatibility:** retain existing higher-priority 
categories and keep unproven/conflicting origins conservative instead of 
redefining all protocol statuses.
   5. **Resource and business ownership:** monitoring releases its references 
without deleting business pending requests or suppressing late callbacks. 
Please assess this bounded lifecycle guarantee rather than a broader Future 
redesign.
   
   ## Compatibility / Impact
   
   - Existing metric names and labels remain. Some failures formerly classified 
as Unknown or ServiceUnavailable can move to Network/Codec when confirmed 
evidence exists; historical category proportions are not guaranteed unchanged.
   - Error text and cause chains are preserved through wrapping and accessible 
via `errors.Is` / `errors.As`; direct assertions on the top-level concrete 
error type are not guaranteed unchanged. Wire status and business-error 
semantics are not rewritten.
   - Metrics count local attempts, not one logical call including retries. 
Attempts do not share evidence.
   - Callback submission no longer means completed success. After unconfirmed 
expiry, processing describes active observations, not proven business 
completion.
   - Once-only publication is not guaranteed delivery: the existing metrics bus 
may drop events when full, so event loss can still cause processing drift.
   - Original user callbacks and Cause behavior remain; no-response callback 
pending retention is not fixed here.
   - Window Gauges reuse current aggregation and do not gain scrape-time 
refresh or automatic idle reset.
   - No Getty upgrade or new race exclusions. Linux/WSL was used because 
existing Windows occupied-port tests have Winsock-specific differences.
   
   ## Scope
   
   ### Included
   
   - Confirmed local Network/Codec origins, request-scoped attribution, and 
corresponding RPC metrics.
   - Dubbo callback monitoring completion/ownership/resource release; atomic 
pending claiming and missing-pending late-response protection.
   - Unit, concurrency, cross-component, loopback-network tests and boundary 
documentation.
   
   ### Not Included
   
   - Complete Provider request-decode/response-encode accounting outside the 
Filter, later Triple streaming-message lifecycles, or new 
compression/decompression classification.
   - Generic Getty connection timeouts that lost their dial cause, or 
connection-wide attribution without reliable RequestID.
   - Complete Dubbo wire-status classification, early business completion, 
no-response pending cleanup, or Future/retry redesign.
   - Full ownership resolution for cancellation versus Reply decoding that has 
already started.
   - Reliable metrics-bus delivery, idle-window refresh, or Getty teardown race 
fixes.
   - Registry-backed full-deployment/retry proof or an Admin browser journey.
   
   ## Related Issue
   
   Related to #3774. No automatic issue-closing keyword is used because the 
broader business-completion concerns are not claimed as resolved.
   
   ## Dependency
   
   No code dependency on #3765. Window metrics reuse the existing aggregation 
interface; idle refresh remains separate.
   
   ## Checklist
   
   - [x] Target the current upstream default branch, `main` (the repository PR 
template still mentions `develop`).
   - [x] Add tests covering the implemented behavior and document known 
boundaries.
   - [x] Check changed Go files with gofmt and the repository imports 
formatter, plus targeted spelling and whitespace checks.
   - [ ] Run full `make fmt` including modernize across the repository; only 
scoped formatting checks were completed locally.
   - [ ] Run full `make test` / `make test-race`; the targeted runs above were 
completed instead.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to