dodjdnh opened a new issue, #3802: URL: https://github.com/apache/dubbo-go/issues/3802
## 1. Problem Currently, dubbo-go records RPC metrics through a **bounded asynchronous event queue** (capacity: 1024). When `Publish` sends an event and the queue is full, the `default` branch is triggered and **silently drops the event**. Under high concurrency, this can severely distort metrics: - **Inaccurate totals**: Even when calls succeed, counter updates, latency samples, and error statistics are directly discarded. - **Drifting concurrency state**: RPC “start” and “completion” events are published independently. Losing either event can cause `processing` (the number of requests being processed) to become negative or retain a positive residual value. > **Load-test observations (1,024 concurrent workers, 3 comparison rounds):** > > The original queue implementation lost thousands of completion events in every round, leaving irreversible `processing` drift ranging from -23 to +10. ## 2. Proposed Solution and References The core direction is to **decouple metric recording from collection and export**, so that the accuracy of core metrics no longer depends on a bounded queue that can drop events. This pattern has been widely validated in **Dubbo Java (3.3)**, Prometheus client_golang, and OpenTelemetry: | Implementation | Recording path | Core mechanism | Risk of queue drops | | --- | --- | --- | --- | | **Current dubbo-go** | Asynchronous enqueueing + background consumption | Bounded Channel (1024) buffers events | **Yes (events are dropped when the queue fills under high concurrency)** | | **Dubbo Java (3.3)** | Direct synchronous dispatch in the call path | `MetricsEventBus` directly invokes listeners, which atomically accumulate local values | **No** (no asynchronous buffer queue) | | **Proposed solution** | Synchronous aggregation in the call path | Cached handles + lazy initialization + in-process atomic accumulation | **No** (direct updates to in-memory state) | ### Key Implementation Points 1. **Synchronous in-place accumulation**: Directly update in-process aggregation state (such as counts and accumulated latency) in the RPC Filter's invocation and return paths, removing the intermediate queue. 2. **Handle caching and lazy binding**: Reuse metric handles based on metadata such as `(Service, Method, Role)` to avoid repeated metric lookups and Tag formatting for every RPC. Dynamically initialize the corresponding series only when a particular error first occurs, avoiding unnecessary memory and high-cardinality overhead. 3. **Preserve existing semantics**: Keep existing metric names, Tag dimensions, the Observation lifecycle, and sliding-window logic unchanged. Network export remains on a separate path. > **Solution diagrams: [dubbo-go Metric Event Loss — Current Implementation and Expected Changes](https://dodjdnh.github.io/genai-docs/docs/dubbo-go-metrics-event-loss/)** > > The diagrams compare the existing asynchronous queue with the proposed synchronous aggregation approach from four perspectives: sequence flow, component relationships, architecture, and memory representation. They also explain the boundaries of the Observation lifecycle, lazy binding, and concurrent updates. <details> <summary>Expand for validation results, questions for maintainers, and additional details (including the Java implementation, code locations, design trade-offs, and reproduction data)</summary> <h2>3. Validation Results (Benchmark)</h2> <p>Comparison using real HTTP/2 RPC between 2 Pods (2 CPU, 2Gi), with 1,024 concurrent workers:</p> <table> <thead> <tr> <th>Metric</th> <th>Existing queue mode</th> <th>Synchronous lazy binding (prototype)</th> <th>Change</th> </tr> </thead> <tbody><tr> <td><strong>Data accuracy</strong></td> <td>1,000–6,000 records missing per round; <code>processing</code> became negative</td> <td><strong>Zero loss</strong>; <code>processing</code> correctly returned to zero</td> <td>Completely resolved missing records</td> </tr> <tr> <td><strong>Throughput (QPS)</strong></td> <td>11,053</td> <td><strong>12,204</strong></td> <td>+10.4%</td> </tr> <tr> <td><strong>P99 latency</strong></td> <td>222.45 ms</td> <td><strong>198.95 ms</strong></td> <td>-10.6%</td> </tr> <tr> <td><strong>Average CPU</strong></td> <td>1.044 / 1.347 cores</td> <td><strong>0.959 / 1.229 cores</strong></td> <td>Approximately 8%–9% lower</td> </tr> </tbody></table> <p><em>Note: Peak memory remained comparable, and all 25 groups of fault-injection and concurrent race checks passed.</em></p> <h2>4. Questions for Maintainers</h2> <ol> <li><strong>Architectural direction</strong>: Does changing built-in metric recording from “asynchronous event delivery” to “synchronous atomic aggregation” align with the project's intended direction?</li> <li><strong>Migration scope and rollout</strong>: Would you support starting with the RPC Filter (PR 1), then gradually evaluating Collectors such as Registry / Config Center?</li> <li><strong>Compatibility boundaries</strong>: For users whose custom extensions directly depend on the <code>metrics.Publish</code> event bus, should a read-only observation bus be retained?</li> </ol> <h2>5. Additional Details</h2> <h3>Corresponding Implementation in Dubbo Java (3.3)</h3> <p>Inspection of the upstream Dubbo Java 3.3 branch shows that its built-in RPC metrics likewise do not use a bounded queue for background consumption:</p> <ol> <li><strong>Request start (MetricsEventBus.before)</strong>: After <code>MetricsFilter</code> creates a <code>RequestEvent</code>, <code>MetricsDispatcher</code> invokes listeners directly through method calls (<code>listener.onEvent</code>), incrementing <code>total</code> and <code>processing</code> in place before proceeding with the RPC. There is no risk of dropping a start event during enqueueing.</li> <li><strong>Request completion (MetricsEventBus.after / error)</strong>: The same <code>RequestEvent</code> saved in the Invocation is used to decrement <code>processing</code> and accumulate results and latency in place in the response or error callback.</li> <li><strong>In-memory storage model (MethodStatComposite)</strong>: Cumulative values are maintained through nested maps of <code>ConcurrentHashMap<MetricType, AtomicLong ConcurrentHashMap<Tags,>></code>. Calling a method one hundred thousand times updates the same set of atomic variables one hundred thousand times, without queueing one hundred thousand events.</li> <li><strong>Limitations</strong>: Dubbo Java addresses “event drops when the queue is full,” but does not provide an absolutely lossless transaction: listener exceptions are only logged, separate field updates expose brief intermediate states, and a process crash before export can still lose data. Synchronous listener execution time also contributes to the call path. This Go proposal borrows its core idea of “direct aggregation without a queue,” then adds handle reuse and memory optimizations suited to Go.</li> </ol> <h3>Relevant Code Locations (dubbo-go)</h3> <ul> <li><a href="https://github.com/apache/dubbo-go/blob/develop/metrics/bus.go"><code>metrics/bus.go</code></a>: <code>Publish</code> drops events without blocking when the queue is full.</li> <li><a href="https://github.com/apache/dubbo-go/blob/develop/metrics/rpc/collector.go"><code>metrics/rpc/collector.go</code></a>: The RPC Channel has a fixed capacity of 1024.</li> <li><a href="https://github.com/apache/dubbo-go/blob/develop/filter/metrics/filter.go"><code>filter/metrics/filter.go</code></a>: Start/End events are published separately, without transactional guarantees.</li> </ul> <h3>Why Not Enlarge the Queue or Use a Blocking Queue?</h3> <ul> <li><strong>Enlarging the queue</strong>: This only absorbs short bursts. Under sustained high load, the queue will still fill and drop data.</li> <li><strong>Blocking queue / unbounded queue</strong>: Blocking propagates metric pressure back to the business RPC path and can overwhelm business processing. An unbounded queue can cause OOM during bursts of errors under high concurrency.</li> <li><strong>Only pairing Start/End</strong>: This can resolve negative <code>processing</code> values, but total call counts and latency bucket distributions are still lost.</li> </ul> <h3>Why Create Metrics Lazily Instead of Pre-binding Everything?</h3> <p>The first prototype preallocated metric series for all failure dimensions. This exposed 34 scalar series per method regardless of whether errors occurred (compared with 18 in the original implementation), causing approximately 20% P99 degradation under concurrent collection. After switching to “atomically initialize and cache handles only when an error first occurs,” overhead became roughly comparable to the original implementation.</p> <h3>Reproduction Data (1,024 Concurrent Workers, 64 RPC Methods)</h3> <table> <thead> <tr> <th>Round</th> <th>Actual completed RPCs</th> <th>Missing Consumer records</th> <th>Missing Provider records</th> <th>Residual Consumer processing</th> <th>Residual Provider processing</th> </tr> </thead> <tbody><tr> <td>1</td> <td>87,118</td> <td>1,070</td> <td>248</td> <td>-4</td> <td>10</td> </tr> <tr> <td>2</td> <td>89,056</td> <td>2,529</td> <td>168</td> <td>-23</td> <td>2</td> </tr> <tr> <td>3</td> <td>93,282</td> <td>6,012</td> <td>416</td> <td>-15</td> <td>4</td> </tr> </tbody></table> </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
