Doris-Breakwater commented on issue #70496: URL: https://github.com/apache/doris/issues/70496#issuecomment-6816732838
Breakwater-GitHub-Analysis-Slot: slot_aaed06a714c1 Initial triage: this warrants priority BE stability investigation because the reported failures repeatedly terminate every BE. The repeated stream assertion is a useful investigation anchor, but the supplied stacks do **not** establish one root cause for all three crash families. Suggested ownership: BE load/BRPC, with storage maintainers reviewing the compaction crash. The issue currently has no labels or assignee. ### What the source confirms I inspected Doris revision `917b20be83878e98f94e3ebf7efccc80346e3e50`, including its third-party patch set. This is source analysis; I have not reproduced these crashes or examined a matching core. * **`Stream::OnReceived()` abort:** this revision uses [BRPC 1.4.0 plus Doris patches](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/thirdparty/vars.sh#L207). A completed DATA frame is submitted to `_consumer_queue`; a nonzero `execution_queue_execute()` result triggers the fatal check. The [Doris background-write patch](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/thirdparty/patches/brpc-1.4.0-write-background.patch) adds three preceding lines, explaining the reported `stream.cpp:470`. In the [queue implementation](https://github.com/apache/brpc/blob/1.4.0/src/bthread/execution_queue_inl.h#L290), failure can mean `EINVAL` for an invalid/stopped queue or `ENOMEM` for allocation failure. The fatal message does not record which occurred. It is **not evidence that an ordinary bounded queue became full**, and it does not establish network failure or compaction overload. * **`Socket::Address(fake_sock_id)` abort:** [stream creation](https://github.com/apache/brpc/blob/1.4.0/src/brpc/stream.cpp#L68) has successfully created a local fake socket and then cannot address its ID. This is a local socket-resource/version invariant failure during stream creation, distinct from receive-queue submission. It is not simply a failed connection to another BE. The complete stack is needed to identify the sender or receiver creation path. * **bthread SIGSEGV:** [`run_main_task()`](https://github.com/apache/brpc/blob/1.4.0/src/bthread/task_group.cpp#L145) obtains a task ID, then [`sched_to()`](https://github.com/apache/brpc/blob/1.4.0/src/bthread/task_group_inl.h#L62) maps it to `TaskMeta` and accesses its stack. An invalid task/metadata pointer is consistent with a low-address fault, including `0x38`, but the function name and fault address alone cannot prove the faulting operand or where the invalid state originated. Registers, disassembly, and the exact overload are needed. * **Compaction SIGSEGV:** [`for_each_tablet()`](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/be/src/storage/tablet/tablet_manager.cpp#L608) copies tablet `shared_ptr`s under the shard lock before invoking the selection callback. Selection accesses tablet metadata/schema, policy and compaction state. The normal tablet ownership path therefore does not by itself demonstrate a dropped-tablet use-after-free. `@0x200` without a source line/instruction cannot identify which object failed. Selection runs even without a large backlog, so low compaction scores neither explain nor exclude this crash. **Still hypotheses:** a stream/queue lifecycle race, earlier memory corruption affecting multiple subsystems, and queue allocation failure remain investigation directions. Their relationship is unverified. Close timestamps on different nodes could reflect shared workload or cascading peer failures, but do not establish causality. `Query id: 0-0` / `tablet id: 0` also cannot identify the triggering SQL or tablet. ### Fix and temporary containment I have not verified a released fix or tested patch for these exact signatures. Importantly, commit ancestry shows that this build **already contains** the receiver close/load-stream race fix [#67080](https://github.com/apache/doris/pull/67080), secondary-package ownership fix [#65819](https://github.com/apache/doris/pull/65819), bthread context-key fix [#67722](https://github.com/apache/doris/pull/67722), and bvar lifetime fix [#68045](https://github.com/apache/doris/pull/68045), as well as the runtime-filter callback fixes #67340/#67912. Reapplying those is not a demonstrated solution. A generic BRPC replacement or removing the assertions would also need lifecycle/ownership validation before production use. A limited, reversible containment experiment is available **if the affected writes use `VTabletWriterV2` load streams**: ```sql SHOW VARIABLES LIKE 'load_stream_per_node'; SET load_stream_per_node = 1; ``` Apply this in one controlled ingestion session first if its current value exceeds 1; save the original value for rollback. The [FE plan builder](https://github.com/apache/doris/blob/917b20be83878e98f94e3ebf7efccc80346e3e50/fe/fe-core/src/main/java/org/apache/doris/qe/runtime/ThriftPlansBuilder.java#L274) passes this value through to the writer's stream pool, so it reduces streams per destination and concurrent stream activity. HTTP Stream Load also accepts a `load_stream_per_node: 1` header in this revision. Confirm the effective count in the `init olap tablet sink` log, and compare throughput, latency, memory and crashes under comparable work. It may reduce throughput and has **no proven crash-prevention effect**; it cannot fix independent scheduler/storage failures. Temporarily limiting concurrent ingestion jobs can likewise reduce this activity, without asserting that the reported workload is excessive. There is no evidence here supporting disabling automatic compaction or increasing `max_tablet_version_num` as a remedy. The inspected metrics do not demonstrate sustained overload, though one sampled tablet cannot characterize the whole cluster. ### Most useful next evidence 1. **Preserve one usable core per distinct signature**, together with the exact crashed executable, build ID and matching debug symbols. For systemd-coredump, collect `coredumpctl info <PID>`, the service's effective `LimitCORE`, coredump size/storage limits and available dump disk space; correct the truncation constraint on a controlled node before the next crash. Do not post raw cores publicly; arrange a private transfer with maintainers. Export `thread apply all bt full`, registers and disassembly at the actual fault frame. For bthread, capture `next_tid`, `next_meta` and task-group state where symbols permit; for compaction, identify the precise object/operand and tablet. 2. **Complete, time-aligned logs from all three BEs and FE audit/error logs**, starting 5–10 minutes before the first crash in each cluster: November 11 around 14:28 and 18:50 CST, plus November 12 around 11:04. Include startup/version headers, full fatal/signal blocks, PID/restart boundaries, and the full `stream.cpp:113` stack. Correlate load IDs, transaction IDs, stream IDs, source/destination BEs, cancellations/timeouts and peer disconnects. Distinguish the first failure from errors after another BE has already exited. 3. **Runtime context:** FE version/commit, distribution/build provenance and compiler, sanitized effective BE configuration and relevant ingestion session/header settings; RSS, available memory, cgroup limits/events, thread counts and kernel OOM/MCE/I/O messages during those windows. The queue's `ENOMEM` path makes contemporaneous allocation evidence useful even with substantial installed RAM. 4. **Workload details:** ingestion methods, concurrent jobs, batch rows/bytes, sanitized SQL/DDL and `EXPLAIN` for overlapping operations; recent schema changes, clone/migration/backup tasks, and any metadata/segment/checksum errors. No deterministic reproduction is required to supply this first diagnostic bundle. For maintainers, the next targeted diagnostic is to preserve the queue return code plus stream/queue IDs and lifecycle state at the failing submission, and fake-socket ID/version/reference state at the creation failure. That distinguishes allocation failure from invalid lifetime/state before selecting a patch. If cores remain inconclusive, reproduce the relevant concurrent workload with a matching ASAN build in a controlled environment to look for the first invalid access rather than treating the final scheduler frame as the root cause. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
