morningman opened a new pull request, #65358:
URL: https://github.com/apache/doris/pull/65358
> [!WARNING]
> **DO NOT MERGE. This is a reproduction/CI branch, not a fix.**
> On an unfixed ASAN build it makes the BE **crash on startup on purpose**.
> It is opened only to run the ASAN pipeline and confirm a stable,
deterministic
> reproduction of a long-standing crash before validating a fix.
## What this is
For years the BE has crashed **only in ASAN builds**, on the thrift RPC
retry/`reopen` path (and, before the `#ifndef ADDRESS_SANITIZER` log guards
were
added, inside `glog` `LOG(WARNING)`), always with the same signature:
```
AddressSanitizer: CHECK failed: asan_thread.cpp:36x
"((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
```
References: internal QA-202 / DORIS-1073 / DORIS-15154, `google/glog#978`,
`google/sanitizers#1010`. It was never root-caused, only worked around by the
scattered `#ifndef ADDRESS_SANITIZER` log guards in `agent/utils.cpp`,
`thrift_rpc_helper.cpp`, `runtime_query_statistics_mgr.cpp`,
`pipeline_fragment_context.cpp`.
## Root cause (summary)
`libthrift` and the other third-party static libs are built **without** ASAN
instrumentation. Under the BE's fully static link (`-static-libstdc++` /
`-static-libasan`) the ASAN interceptors for `__cxa_throw` /
`_Unwind_RaiseException` are **bypassed by a strong/weak symbol collision**,
so
`__asan_handle_no_return()` never runs when un-instrumented code throws (e.g.
`TSocket` throwing `TTransportException` when the FE restarts). The unwound
instrumented frames keep their **stack shadow poison**; the retry path
immediately re-descends and thrift/glog put innocent stack locals on that
stale
poison. The next intercepted libc call (`memcpy`/`memset`/`localtime_r`/…)
raises
a **false positive**, and ASAN aborts while walking the already-overwritten
frame
descriptor — hence `CHECK failed ... (0x0, 0x0)` instead of a normal report.
This cannot happen in RELEASE/DEBUG builds (no shadow memory; the accessed
memory is entirely valid).
## What the reproducer does
Runs once at BE startup, **ASAN builds only** (no-op otherwise; skip with
`DORIS_DISABLE_ASAN_REPRO=1`):
1. **part 1** — measures the stale poison deterministically with
`__asan_region_is_poisoned()`, plus a control run that throws from
*instrumented* code to show the asymmetry (the compiler emits
`__asan_handle_no_return()` only for instrumented throws).
2. **part 2** — drives the **real Jira stack site**: a real
`ThriftClient<FrontendServiceClient>` `report()` against an in-process
dead
peer that RSTs the connection, then `ThriftClientImpl::close()` →
`TBufferedTransport::close/flush` → `TSocket::write_partial` →
`getSocketInfo()`, swept over 257 stack offsets so the poison/local
overlap is
hit deterministically.
3. **part 3** — synthetic fallback crash site if part 2 gets lucky.
`be/src/common/asan_repro_uninstrumented.cpp` is compiled with
`-fno-sanitize=all -fno-builtin` to emulate an un-instrumented third-party
lib.
## How to read the CI output (grep `[ASAN-REPRO]` in `be.out`)
- `... __cxa_throw ... => NOT intercepted (bypassed by static link)` —
proves the bypass.
- `part1 test (throw from UN-INSTRUMENTED code): N/160 ...` with `N > 0` —
stale poison confirmed.
- Crash with frames `getSocketInfo ← write_partial ← flush ←
TBufferedTransport::close ← ThriftClientImpl::close` — reproduced at the
original Jira site.
- `RESULT: NEGATIVE ... BE continues to start.` — the bug is **not** present
(this is the signal a candidate fix works).
## Candidate fix (separate PR, not here)
Link the ASAN build with `-Wl,--wrap=__cxa_throw` (+ `__cxa_rethrow`) and
forward
through `__asan_handle_no_return()`; or drop `-static-libstdc++` for the ASAN
build; or build an instrumented third-party set for the ASAN pipeline.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
