morningman opened a new pull request, #65366:
URL: https://github.com/apache/doris/pull/65366
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #65358 (reproduction & verification branch, do-not-merge),
workaround history: #34813 #34994 #36808 #37134 #48347 #49516 #49878
Problem Summary:
For years the BE has crashed **only in ASAN builds**, on the thrift RPC
retry/`reopen` path (and, before the `#ifndef ADDRESS_SANITIZER` log guards
were added, inside glog `LOG(WARNING)`), always with the same signature:
```
AddressSanitizer: CHECK failed: asan_thread.cpp:36x
"((ptr[0] == kCurrentStackFrameMagic)) != (0)" (0x0, 0x0)
```
Same failure family as google/glog#978 and google/sanitizers#1010. It was
never root-caused, only worked around with ASAN-only log guards and code-path
forks — and #49516 showed those forks hatch real bugs of their own.
**Root cause.** libthrift and the other third-party static libs are built
**without** ASAN instrumentation. ASAN cleans the stack shadow on exception
unwinding via `__asan_handle_no_return()`: the compiler emits it before
instrumented throws, and the runtime provides weak `__cxa_throw` /
`_Unwind_RaiseException` interceptors to cover un-instrumented throws. Under
the BE's fully static link (`-static-libstdc++` / `-static-libgcc` plus a
static sanitizer runtime) those **weak** interceptors lose to libstdc++.a's /
libgcc_eh.a's **strong** definitions at link time, so the cleanup never runs
when un-instrumented code throws (e.g. `TSocket` raising `TTransportException`
when the FE restarts). The unwound instrumented frames keep their stack-shadow
poison; the retry path immediately re-descends over the same stack range,
innocent locals (thrift's `getSocketInfo()` strings, glog's `struct tm`, ...)
land on the stale poison, the first intercepted write reports a false positive,
and ASAN a
borts while walking the already-overwritten frame descriptor — before it can
even print a normal report. RELEASE/DEBUG builds have no shadow memory, which
is why this never happened outside ASAN.
**The fix.** ASAN/ASAN_UT Linux links now pass
```
-Wl,--wrap=__cxa_throw -Wl,--wrap=__cxa_rethrow
-Wl,--wrap=_Unwind_RaiseException
```
and `be/src/common/asan_cxa_throw_wrap.cpp` provides wrappers that call
`__asan_handle_no_return()` before forwarding to the real implementations —
exactly what the bypassed official interceptors do. `--wrap` rewrites every
reference from every input object, including the un-instrumented third-party
archives and libstdc++.a itself. The flags are attached to `DORIS_LINK_LIBS`,
so any executable receiving them also links `Common`, which carries the wrapper
definitions — flags and definitions cannot drift apart.
`_Unwind_RaiseException` is wrapped too because `std::rethrow_exception` and
foreign-language unwinders raise through it without touching `__cxa_throw`; its
wrapper forwards the return value (it returns when no handler is found).
**Production (RELEASE/DEBUG) builds are byte-identical.**
**Cleanup.** With the throw path fixed, the ASAN-only forks are removed — 15
`#ifndef ADDRESS_SANITIZER` sites in `thrift_rpc_helper.cpp` (6),
`agent/utils.cpp` (6), `runtime_query_statistics_mgr.cpp` (2, one of which
skipped the whole reopen+retry under ASAN, the failure pattern #49516 exposed),
`pipeline_fragment_context.cpp` (1), plus the leftover `std::cerr` LOG
substitutes. ASAN CI now exercises the exact production retry/reopen/logging
code.
**Verification.** Done on the ASAN pipeline via the dedicated reproduction
branch #65358, which carries a deterministic startup reproducer (not part of
this PR):
| | unfixed (baseline) | with this fix |
|---|---|---|
| interceptor check | `__cxa_throw ... NOT intercepted (bypassed by static
link)` | `WRAPPED by __wrap___cxa_throw` |
| stale shadow poison after an un-instrumented throw | 58/160 probed frames
poisoned | 0/160 |
| driving the historical crash site (`report()` fail → `close()` → `flush()`
→ `getSocketInfo()`, swept over 257 stack offsets) | deterministic startup
crash with the historical stack | survives the full sweep, BE starts and runs
the regression |
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- A/B verified on the ASAN pipeline via the deterministic reproducer
branch #65358 (see the verification table above): the reproducer crashes every
unfixed ASAN build at startup and passes with this fix, after which the full
ASAN regression runs normally.
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
