matanper opened a new issue, #70489: URL: https://github.com/apache/doris/issues/70489
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version 4.1.4. master (`9e598d38789`) has the same code. ### What's Wrong? A storage Top-N read (`VCollectIterator::_topn_next`) over a rowset whose segments overlap fails with `[E-7412]` (a column type mismatch while appending rows) when the scan has virtual columns: an ANN index-only distance column, or a virtual slot pushed down by common-subexpression elimination. Without overlapping segments, or without virtual columns, the same query works. ### Why `_topn_next` (`be/src/storage/iterator/vcollect_iterator.cpp:284`) keeps its Top-N candidates in `clone_block = block->clone_empty()` and gives that clone real columns for the virtual slots (`:296-303`). The original `block` is then reused as the per-rowset read buffer, and its virtual slots are still `ColumnNothing` placeholders. - A rowset with non-overlapping segments is read by the segment iterator, which fills virtual slots itself. - A rowset with overlapping segments goes through the ordered merge iterator. It appends already-materialized values into the caller's columns (`be/src/storage/iterator/vgeneric_iterators.cpp:152`, `insert_range_from`). For a virtual slot that destination is the `ColumnNothing` placeholder, whose `insert_range_from` only adds to the row count and drops the values (`be/src/core/column/column_dummy.h:83-85`). - The rows are then copied into the typed accumulator with `MutableBlock::add_rows` (`be/src/core/block/block.cpp:1150`/`:1170`), which fails on the `ColumnNothing` → typed column mismatch. ### What You Expected? The Top-N query returns the correct rows and virtual-column values, whether or not the rowset's segments overlap. ### How to Reproduce? 1. A duplicate-key table with an ANN index (or any query whose plan pushes a virtual column into the OLAP scan) and a sort key. 2. Load data so that one rowset has overlapping segments (several segments in one rowset whose key ranges overlap, for example one large load). 3. Query it with a Top-N on the sort key that reads the virtual column, for example: ```sql SELECT id, l2_distance_approximate(embedding, [..]) AS d FROM t WHERE l2_distance_approximate(embedding, [..]) < 10 ORDER BY id LIMIT 20; ``` 4. The query fails with `[E-7412]`. A BE unit test that runs the real merge iterator over two overlapping in-memory segment iterators with a virtual column through `VCollectIterator::_topn_next` reproduces it. ### Anything Else? Proposed fix: give the read `block`'s virtual slots their real columns before `clone_empty()`, instead of only the clone's. Both the merge destination and the accumulator are then typed, and the clone inherits the types. Rowsets read by the segment iterator are unaffected, since it resets and fills virtual slots itself. With no virtual columns the change does nothing. We have a patch with a regression test and can open a PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
