dodjdnh opened a new issue, #3802:
URL: https://github.com/apache/dubbo-go/issues/3802

   ## 1. Problem
   
   Currently, dubbo-go records RPC metrics through a **bounded asynchronous 
event queue** (capacity: 1024). When `Publish` sends an event and the queue is 
full, the `default` branch is triggered and **silently drops the event**.
   
   Under high concurrency, this can severely distort metrics:
   
   - **Inaccurate totals**: Even when calls succeed, counter updates, latency 
samples, and error statistics are directly discarded.
   - **Drifting concurrency state**: RPC “start” and “completion” events are 
published independently. Losing either event can cause `processing` (the number 
of requests being processed) to become negative or retain a positive residual 
value.
   
   > **Load-test observations (1,024 concurrent workers, 3 comparison rounds):**
   >
   > The original queue implementation lost thousands of completion events in 
every round, leaving irreversible `processing` drift ranging from -23 to +10.
   
   ## 2. Proposed Solution and References
   
   The core direction is to **decouple metric recording from collection and 
export**, so that the accuracy of core metrics no longer depends on a bounded 
queue that can drop events. This pattern has been widely validated in **Dubbo 
Java (3.3)**, Prometheus client_golang, and OpenTelemetry:
   
   | Implementation | Recording path | Core mechanism | Risk of queue drops |
   | --- | --- | --- | --- |
   | **Current dubbo-go** | Asynchronous enqueueing + background consumption | 
Bounded Channel (1024) buffers events | **Yes (events are dropped when the 
queue fills under high concurrency)** |
   | **Dubbo Java (3.3)** | Direct synchronous dispatch in the call path | 
`MetricsEventBus` directly invokes listeners, which atomically accumulate local 
values | **No** (no asynchronous buffer queue) |
   | **Proposed solution** | Synchronous aggregation in the call path | Cached 
handles + lazy initialization + in-process atomic accumulation | **No** (direct 
updates to in-memory state) |
   
   ### Key Implementation Points
   
   1. **Synchronous in-place accumulation**: Directly update in-process 
aggregation state (such as counts and accumulated latency) in the RPC Filter's 
invocation and return paths, removing the intermediate queue.
   2. **Handle caching and lazy binding**: Reuse metric handles based on 
metadata such as `(Service, Method, Role)` to avoid repeated metric lookups and 
Tag formatting for every RPC. Dynamically initialize the corresponding series 
only when a particular error first occurs, avoiding unnecessary memory and 
high-cardinality overhead.
   3. **Preserve existing semantics**: Keep existing metric names, Tag 
dimensions, the Observation lifecycle, and sliding-window logic unchanged. 
Network export remains on a separate path.
   
   > **Solution diagrams: [dubbo-go Metric Event Loss — Current Implementation 
and Expected 
Changes](https://dodjdnh.github.io/genai-docs/docs/dubbo-go-metrics-event-loss/)**
   >
   > The diagrams compare the existing asynchronous queue with the proposed 
synchronous aggregation approach from four perspectives: sequence flow, 
component relationships, architecture, and memory representation. They also 
explain the boundaries of the Observation lifecycle, lazy binding, and 
concurrent updates.
   
   <details>
   <summary>Expand for validation results, questions for maintainers, and 
additional details (including the Java implementation, code locations, design 
trade-offs, and reproduction data)</summary>
   <h2>3. Validation Results (Benchmark)</h2>
   <p>Comparison using real HTTP/2 RPC between 2 Pods (2 CPU, 2Gi), with 1,024 
concurrent workers:</p>
   <table>
   <thead>
   <tr>
   <th>Metric</th>
   <th>Existing queue mode</th>
   <th>Synchronous lazy binding (prototype)</th>
   <th>Change</th>
   </tr>
   </thead>
   <tbody><tr>
   <td><strong>Data accuracy</strong></td>
   <td>1,000–6,000 records missing per round; <code>processing</code> became 
negative</td>
   <td><strong>Zero loss</strong>; <code>processing</code> correctly returned 
to zero</td>
   <td>Completely resolved missing records</td>
   </tr>
   <tr>
   <td><strong>Throughput (QPS)</strong></td>
   <td>11,053</td>
   <td><strong>12,204</strong></td>
   <td>+10.4%</td>
   </tr>
   <tr>
   <td><strong>P99 latency</strong></td>
   <td>222.45 ms</td>
   <td><strong>198.95 ms</strong></td>
   <td>-10.6%</td>
   </tr>
   <tr>
   <td><strong>Average CPU</strong></td>
   <td>1.044 / 1.347 cores</td>
   <td><strong>0.959 / 1.229 cores</strong></td>
   <td>Approximately 8%–9% lower</td>
   </tr>
   </tbody></table>
   <p><em>Note: Peak memory remained comparable, and all 25 groups of 
fault-injection and concurrent race checks passed.</em></p>
   <h2>4. Questions for Maintainers</h2>
   <ol>
   <li><strong>Architectural direction</strong>: Does changing built-in metric 
recording from “asynchronous event delivery” to “synchronous atomic 
aggregation” align with the project's intended direction?</li>
   <li><strong>Migration scope and rollout</strong>: Would you support starting 
with the RPC Filter (PR 1), then gradually evaluating Collectors such as 
Registry / Config Center?</li>
   <li><strong>Compatibility boundaries</strong>: For users whose custom 
extensions directly depend on the <code>metrics.Publish</code> event bus, 
should a read-only observation bus be retained?</li>
   </ol>
   <h2>5. Additional Details</h2>
   <h3>Corresponding Implementation in Dubbo Java (3.3)</h3>
   <p>Inspection of the upstream Dubbo Java 3.3 branch shows that its built-in 
RPC metrics likewise do not use a bounded queue for background consumption:</p>
   <ol>
   <li><strong>Request start (MetricsEventBus.before)</strong>: After 
<code>MetricsFilter</code> creates a <code>RequestEvent</code>, 
<code>MetricsDispatcher</code> invokes listeners directly through method calls 
(<code>listener.onEvent</code>), incrementing <code>total</code> and 
<code>processing</code> in place before proceeding with the RPC. There is no 
risk of dropping a start event during enqueueing.</li>
   <li><strong>Request completion (MetricsEventBus.after / error)</strong>: The 
same <code>RequestEvent</code> saved in the Invocation is used to decrement 
<code>processing</code> and accumulate results and latency in place in the 
response or error callback.</li>
   <li><strong>In-memory storage model (MethodStatComposite)</strong>: 
Cumulative values are maintained through nested maps of 
<code>ConcurrentHashMap&lt;MetricType, AtomicLong 
ConcurrentHashMap&lt;Tags,&gt;&gt;</code>. Calling a method one hundred 
thousand times updates the same set of atomic variables one hundred thousand 
times, without queueing one hundred thousand events.</li>
   <li><strong>Limitations</strong>: Dubbo Java addresses “event drops when the 
queue is full,” but does not provide an absolutely lossless transaction: 
listener exceptions are only logged, separate field updates expose brief 
intermediate states, and a process crash before export can still lose data. 
Synchronous listener execution time also contributes to the call path. This Go 
proposal borrows its core idea of “direct aggregation without a queue,” then 
adds handle reuse and memory optimizations suited to Go.</li>
   </ol>
   <h3>Relevant Code Locations (dubbo-go)</h3>
   <ul>
   <li><a 
href="https://github.com/apache/dubbo-go/blob/develop/metrics/bus.go";><code>metrics/bus.go</code></a>:
 <code>Publish</code> drops events without blocking when the queue is full.</li>
   <li><a 
href="https://github.com/apache/dubbo-go/blob/develop/metrics/rpc/collector.go";><code>metrics/rpc/collector.go</code></a>:
 The RPC Channel has a fixed capacity of 1024.</li>
   <li><a 
href="https://github.com/apache/dubbo-go/blob/develop/filter/metrics/filter.go";><code>filter/metrics/filter.go</code></a>:
 Start/End events are published separately, without transactional 
guarantees.</li>
   </ul>
   <h3>Why Not Enlarge the Queue or Use a Blocking Queue?</h3>
   <ul>
   <li><strong>Enlarging the queue</strong>: This only absorbs short bursts. 
Under sustained high load, the queue will still fill and drop data.</li>
   <li><strong>Blocking queue / unbounded queue</strong>: Blocking propagates 
metric pressure back to the business RPC path and can overwhelm business 
processing. An unbounded queue can cause OOM during bursts of errors under high 
concurrency.</li>
   <li><strong>Only pairing Start/End</strong>: This can resolve negative 
<code>processing</code> values, but total call counts and latency bucket 
distributions are still lost.</li>
   </ul>
   <h3>Why Create Metrics Lazily Instead of Pre-binding Everything?</h3>
   <p>The first prototype preallocated metric series for all failure 
dimensions. This exposed 34 scalar series per method regardless of whether 
errors occurred (compared with 18 in the original implementation), causing 
approximately 20% P99 degradation under concurrent collection. After switching 
to “atomically initialize and cache handles only when an error first occurs,” 
overhead became roughly comparable to the original implementation.</p>
   <h3>Reproduction Data (1,024 Concurrent Workers, 64 RPC Methods)</h3>
   <table>
   <thead>
   <tr>
   <th>Round</th>
   <th>Actual completed RPCs</th>
   <th>Missing Consumer records</th>
   <th>Missing Provider records</th>
   <th>Residual Consumer processing</th>
   <th>Residual Provider processing</th>
   </tr>
   </thead>
   <tbody><tr>
   <td>1</td>
   <td>87,118</td>
   <td>1,070</td>
   <td>248</td>
   <td>-4</td>
   <td>10</td>
   </tr>
   <tr>
   <td>2</td>
   <td>89,056</td>
   <td>2,529</td>
   <td>168</td>
   <td>-23</td>
   <td>2</td>
   </tr>
   <tr>
   <td>3</td>
   <td>93,282</td>
   <td>6,012</td>
   <td>416</td>
   <td>-15</td>
   <td>4</td>
   </tr>
   </tbody></table>
   </details>
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to