bosswnx opened a new issue, #67428: URL: https://github.com/apache/doris/issues/67428
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues) and found no similar issues. ### Version master. Reproduced on two independent runs with bases `5fbe3600024` (PR #67404) and `952bfcbb40fa` (PR #67402), both containing #66899 (merged Sep 1 15:33) and #66191 (merged Sep 1 18:24). The use-after-free itself was introduced by `e925e6ae1b2` (#66889, merged Aug 27). ### What's Wrong? Two unrelated test-only PRs (#67404, #67402 — neither touches BE code) both hit the **same** BE crash in the nonConcurrent regression pipeline: an ASAN heap-use-after-free that kills the BE process, after which ~49 suites fail in cascade ("replication num ... available backend num is 0") and the whole pipeline goes FAILURE. Crash from PR #67402's build (1036154) — identical stack in PR #67404's build (1035988): ``` ==48480==ERROR: AddressSanitizer: heap-use-after-free on address 0x7d3e73b61f30 READ of size 8 at 0x7d3e73b61f30 thread T1919 (mf_normal [work) #0 std::__shared_ptr<doris::segment_v2::SegmentAllocatedLsnMap>::operator bool() const #2 doris::RowsetWriterContext::remove_segment_allocated_lsns(long) be/src/storage/rowset/rowset_writer_context.h:202:9 #3 doris::SharedMemtable::~SharedMemtable() be/src/load/memtable/memtable_flush_executor.cpp:124:21 #7 doris::PartOfGroupMemtableFlushTask::~PartOfGroupMemtableFlushTask() be/src/load/memtable/memtable_flush_executor.cpp:82:7 #13 doris::ThreadPool::dispatch_thread() be/src/util/threadpool.cpp:628:23 freed by thread T1919 (mf_normal [work) here: #4 doris::FlushToken::~FlushToken() be/src/load/memtable/memtable_flush_executor.h:92:7 #8 doris::PartOfGroupMemtableFlushTask::run() be/src/load/memtable/memtable_flush_executor.cpp:100:5 previously allocated by thread T640 (brpc_heavy) here: #1 doris::RowsetFactory::create_empty_group_rowset_writer(...) be/src/storage/rowset/rowset_factory.cpp:100 #2 doris::GroupRowsetBuilder::init() be/src/storage/rowset_builder.cpp:570:5 #4 doris::DeltaWriter::write(...) be/src/load/delta_writer/delta_writer.cpp:177:9 #15 doris::PInternalService::tablet_writer_add_block ... (stream load / group commit ingest path) SUMMARY: AddressSanitizer: heap-use-after-free be/src/storage/rowset/rowset_writer_context.h:202:9 in doris::RowsetWriterContext::remove_segment_allocated_lsns(long) ``` **Mechanism.** At the end of `PartOfGroupMemtableFlushTask::run()` (`memtable_flush_executor.cpp:100`), the local `shared_ptr<FlushToken>` is destroyed. When it holds the last reference, destruction cascades `FlushToken` → `_rowset_writer` → `RowsetWriter` → `RowsetWriterContext`. Afterwards the thread pool (`dispatch_thread`, threadpool.cpp:628) destroys the task object itself; `~PartOfGroupMemtableFlushTask` → `~SharedMemtable` (memtable_flush_executor.cpp:124) then calls `remove_segment_allocated_lsns()` through the **raw pointer** `SharedMemtable::rowset_ctx` (`memtable_flush_executor.h:67`), which already dangles. The raw `rowset_ctx` and `has_allocated_lsns` were added by #66889 (`git log -S "has_allocated_lsns"` → `e925e6ae1b2`). **Timeline** (all times UTC+8, from commit statuses of recent nonConcurrent runs): | Time | Event | |---|---| | Aug 27 12:07 | #66889 merged (UAF code enters master) | | Aug 31 – Sep 1 16:25 | all nonConcurrent runs pass (e.g. #67344, #67341 — bases without #66899/#66191) | | Sep 1 15:33 | #66899 merged (row binlog flexible partial updates; also adds 2 binlog regression suites) | | Sep 1 18:24 | #66191 merged (binlog tablets colocated — rewrites `GroupRowsetBuilder::init` in `rowset_builder.cpp`) | | Sep 1 19:09 | `5fbe3600024` = base of #67404 | | Sep 1 19:56 | `952bfcbb40fa` = base of #67402 | | Sep 2 03:59 / 12:50 | #67404 (build 1035988) and #67402 (build 1036154) both crash with the identical UAF | So the UAF lay dormant for ~5 days and became reproducible once #66191/#66899 landed (they change object lifetimes / widen coverage on the group-commit + row-binlog write path). Any PR based on master ≥ Sep 1 evening currently fails nonConcurrent. ### What You Expected? The nonConcurrent regression should pass on master. A group-commit memtable flush task must not access a `RowsetWriterContext` after the `FlushToken`'s last reference is released. ### How to Reproduce? 1. Create a PR based on master ≥ `5fbe3600024` (any change; the two hits were test-only PRs). 2. Comment `run buildall` to trigger the nonConcurrent pipeline. 3. During the first group-commit / stream-load ingest suites, BE exits with the ASAN heap-use-after-free above; subsequent suites fail with "available backend num is 0". Reference builds: #67404 → TeamCity build 1035988; #67402 → TeamCity build 1036154. ### Anything Else? Suggested fix directions (either one): 1. `SharedMemtable` holds a `std::shared_ptr<RowsetWriter>` (or directly the `std::shared_ptr<SegmentAllocatedLsnMap>`) instead of the raw `RowsetWriterContext*`, so the LSN map outlives the flush task. 2. Reorder destruction inside `PartOfGroupMemtableFlushTask::run()` so `_shared_memtable` is released before the local `FlushToken` shared_ptr goes out of scope (e.g. an explicit scope block). CC the authors of the related PRs: #66889 (introduced the raw pointer + LSN bookkeeping), #66191 / #66899 (landed in the window where this became reproducible). ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
