Doris-Breakwater commented on issue #66390: URL: https://github.com/apache/doris/issues/66390#issuecomment-5166245921
## Initial maintainer analysis **Triage:** This is a credible high-severity BE crash. The issue currently has no labels; `bug`, `BE/storage`, and the project's high-priority label would be appropriate. I found a concrete code-level crash condition at the reported commit that closely matches the stack, but the missing SQL/core operands prevent attributing this particular incident with certainty. ### Verified code facts - `VMergeIterator::init()` initializes each context, skips invalid/EOF contexts, and then pushes a `shared_ptr` into the heap. This makes a simple context-lifetime or "uninitialized context was pushed" explanation unlikely: [vgeneric_iterators.cpp#L294-L303](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/storage/iterator/vgeneric_iterators.cpp#L294-L303), [vgeneric_iterators.cpp#L357-L373](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/storage/iterator/vgeneric_iterators.cpp#L357-L373). - Every ROW-binlog scan forces a key-ordered read, sets the order-by prefix count to zero, and can force a per-rowset merge even for non-overlapping segments: [olap_scanner.cpp#L568-L578](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/exec/scan/olap_scanner.cpp#L568-L578), [beta_rowset_reader.h#L62-L65](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/storage/rowset/beta_rowset_reader.h#L62-L65). With a zero prefix, `_compare_columns` remains null, so the comparator takes the exact line-128 branch and compares `_num_key_columns`. - `VMergeIteratorContext::_num_key_columns` comes from `_output_schema->num_key_columns()`: [vgeneric_iterators.h#L86-L98](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/storage/iterator/vgeneric_iterators.h#L86-L98). However, `Schema(columns, col_ids)` counts key columns across the **entire tablet schema**, not across the projected `col_ids`: [schema.h#L51-L81](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/storage/schema.h#L51-L81). - For `MIN_DELTA`/`DETAIL`, the scanner explicitly widens the storage projection with all key columns. For other binlog modes, notably `APPEND_ONLY`, the `direct_mode` branch can leave `return_columns` as only the SQL projection: [olap_scanner.cpp#L471-L525](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/exec/scan/olap_scanner.cpp#L471-L525). This is reachable for the normal no-storage-merge DUP/UNIQUE-MOW path. - `Block::compare_at()` only has `DCHECK` bounds checks and then uses unchecked positional access. In a release BE, `_num_key_columns > block.columns()` can therefore dereference past the block's column array instead of returning a `Status`: [block.h#L375-L385](https://github.com/apache/doris/blob/1590e7e3228f8f0cafcaa89e841fece09c2475be/be/src/core/block/block.h#L375-L385). ### Leading root-cause hypothesis (strong, not incident-confirmed) An `APPEND_ONLY` ROW-binlog query projected fewer columns than the tablet's key count, or projected columns without all leading keys. The output block was built from that projection, while the merge comparator retained the full tablet key count. Once a rowset with at least two segments was initialized, the second heap insertion called `compare()` and read a missing/wrong positional column. This explains all distinctive evidence in the report: 1. ROW-binlog workload; 2. `VMergeIterator::init()` / `std::push_heap()` rather than `_next_batch()`; 3. line 128, the default key-column comparison branch; 4. a release-build `SIGSEGV` rather than a diagnostic error. Even when `block.columns() >= _num_key_columns`, a projection that omits/reorders keys can make the comparator sort by value columns, so this is also a correctness invariant issue, not only an out-of-bounds issue. This mechanism is different from #16495, whose fix corrected a sequence-column schema-ID/block-position mismatch. ### Evidence still needed to confirm this crash - The original SQL or at least its `incrementType` and projected columns, plus `EXPLAIN VERBOSE`. - `SHOW CREATE TABLE`/full schema, including key type, key columns, ROW-binlog properties, and hidden binlog columns. - The full signal header (query ID included), relevant BE log interval, and the scan tablet/rowset ID. - Rowset metadata: segment count, segment overlap flag, and selected segment offsets. - At the crashing frame, for both operands: `_valid`, `_index_in_block`, `_block`, `_block->rows()`, `_block->columns()`, `_num_key_columns`, `_compare_columns`, `_sequence_id_idx`, `_output_schema->column_ids()`, and `_output_schema->num_column_ids()`. The decisive check is whether either block has fewer columns than `_num_key_columns`, or whether its first `_num_key_columns` positions are not the full ordered key prefix. ### Recommended next steps 1. First validate the above values in the retained core. This can confirm or reject the projected-schema hypothesis without recovering the whole SQL. 2. Minimize from the existing ROW-binlog regression tests with an `APPEND_ONLY` query that omits one or all key columns and a rowset forced to contain at least two segments. Cover both DUP and UNIQUE-MOW where practical. 3. Fix the invariant at its source: every path that requests/forces key-ordered merging must supply the full key prefix to the storage projection, while the existing output projection removes those internal columns before returning rows. Do not only catch the comparator failure, because comparing projected value columns would still silently misorder rows. 4. Add an init-time validation that all key/explicit comparison/sequence positions exist in the blocks and return an error containing tablet, rowset, segment, projection, and comparison-index details if the contract is violated. 5. Add a focused BE unit test using a projected output schema plus two merge inputs, and a regression test for ROW-binlog `APPEND_ONLY` with a key-omitting projection and a multi-segment rowset. Breakwater-GitHub-Analysis-Slot: slot_2156593f0d29 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
