Doris-Breakwater commented on issue #69724: URL: https://github.com/apache/doris/issues/69724#issuecomment-6461946093
Initial triage: the reported configuration matrix strongly implicates the single-replica Stream Load path. Treat this as a suspected 4.1.5 regression in loading/RPC completion, pending a controlled reproduction and stacks. It does **not yet establish** a dropped batch, a broken brpc socket, or a specific deadlock. The captured issue has no labels; suggested classification is bug / loading, with regression tracking once confirmed. I compared the exact reported commits, `ad35a140c7f` and `917b20be838`, in the source repository. This was static inspection; I have not reproduced the failure. **What the code establishes** - `VNodeChannel::mark_close()` queues the final EOS batch. For a small load, “left pending batch size: 1” is expected. `try_send_pending_block()` sets `_send_finished` when preparing that final RPC, and the sender loop can then exit while its completion is outstanding. The two quoted messages therefore do not establish that the batch was discarded. See [the sender and EOS handling at the reported commit](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/exec/sink/writer/vtablet_writer.cpp#L1123). - The `VNodeChannel` signature identifies the `VTabletWriter` batch-RPC path. Its receiver chain is `tablet_writer_add_block` → `LoadChannelMgr::add_batch` → `LoadChannel::add_batch` → `TabletsChannel::close`. After flushing/building and committing the primary rowset, `DeltaWriter` requests secondary replicas to pull it. That adds **brpc requests to the secondaries, HTTP file downloads back from the primary's advertised HTTP endpoint, and completion RPCs back to the primary**. Successful TCP probes and brpc `/status` responses do not exercise that entire sequence. See [primary commit and pull request construction](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/load/delta_writer/delta_writer.cpp#L240) and [secondary download handling](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/service/internal_service.cpp#L1968). - `TabletsChannel::close()` waits for secondary completion, bounded by approximately 90% of the load-channel timeout **after reaching that wait**. The enclosing batch RPC has its own deadline. The reported `599998ms` error is evidence of an approximately ten-minute RPC timeout, not evidence that every wait is literally unbounded. HTTP `100 Continue` is interim; it does not indicate load completion. See [receiver close/wait logic](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/load/channel/tablets_channel.cpp#L375). - SQL `INSERT` success is a useful control, but does not prove that it exercised the same BE write path. Stream Load's single-replica decision depends on the effective memtable-on-sink setting; SQL INSERT uses a separate `enable_single_replica_insert` session setting. See [Stream Load selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/load/NereidsLoadPlanInfoCollector.java#L305) and [INSERT selection](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalOlapTableSinkToPhysicalOlapTableSink.java#L35). **Hypotheses that still need evidence** The current evidence cannot distinguish an outstanding batch RPC before receiver execution from a receiver blocked in memory admission, flush/commit, secondary HTTP download, or completion processing. Searching a peer's INFO logs only by transaction ID is insufficient: several normal-path messages are verbose-only, and batch requests are identified by load ID. There is a known starvation mechanism documented publicly in [#42297](https://github.com/apache/doris/pull/42297): close requests occupy the worker pool needed by secondary completion requests. In **both exact builds here**, `tablet_writer_add_block` and `response_slave_tablet_pull_rowset` use `_heavy_work_pool`. This is a diagnostic lead, not an explanation of the version difference. A single serial load with that pool idle throughout the stall would argue against it. An empty pool before reproduction, or absence of queue-rejection logs, does not establish its state during the stall. Increasing pool sizes without this evidence is premature. The related reports are not confirmed duplicates. [#65582](https://github.com/apache/doris/issues/65582) concerns V2 incremental load-stream close handling, whereas these logs identify the batch-RPC writer; [#64708](https://github.com/apache/doris/issues/64708) needs matching socket evidence; [#68973](https://github.com/apache/doris/issues/68973) describes HTTP worker blocking that could amplify a stalled load without identifying its original cause. **Information needed for the next step** 1. A minimal `SHOW CREATE TABLE`, one-row payload, and complete curl command with credentials removed. Preserve redirect handling, all load headers, `Expect` behavior, explicit server `timeout`, and client timeout. Include timestamped verbose output confirming the body was sent. State whether one serial request on an otherwise idle cluster hangs immediately after restart. 2. Effective FE/BE configuration and build IDs from every participating process, plus Stream Load's `memtable_on_sink_node`, group-commit settings, table light-schema-change property, and relevant SQL INSERT session settings. The sibling-cluster comparison is valuable, but a same-environment A/B reproduction is needed to isolate version from configuration/platform differences. 3. For one hanging load, synchronized FE and **all BE** logs from transaction begin through timeout, correlated by label, load ID, transaction ID, tablet ID, and selected primary/secondary backend IDs. In particular, locate receiver “close tablets channel,” pull/download errors, and secondary completion messages. Establish whether the quoted receiver is the selected primary or a secondary. 4. Two or three full pstack/thread dumps, several seconds apart, from the coordinator, primary and secondaries **while the request is stalled**. Capture contemporaneous heavy-pool active/queued counts and brpc method/socket statistics, including the advertised primary HTTP address. These distinguish pre-handler transport stalls, flush/commit waits, HTTP download waits, and completion starvation. Capture before another restart. For mitigation, keep the reported working combination—FE single-replica load disabled and BE `enable_single_replica_load=false`—while preserving `replication_num=3`. Disabling only the BE switch produces the expected rejection already demonstrated in the matrix. Reducing replication or increasing RPC timeouts would not resolve the mechanism. Maintainer next step: reproduce the one-row case with explicit server/client deadlines, identify the last completed boundary above, then compare/bisect the two exact revisions in a test environment. The sender implementation differs only by a cancellation comment between these builds; shared RPC callback/profile code also changed, so the log wording alone cannot identify an introducing commit. No specific fix or fixed release is established by this investigation. Breakwater-GitHub-Analysis-Slot: slot_24ddcce191e8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
