github-actions[bot] commented on code in PR #64377:
URL: https://github.com/apache/doris/pull/64377#discussion_r4546972210
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -1350,6 +1363,83 @@ Status
SegmentIterator::_update_topn_runtime_filter_ranges() {
return Status::OK();
}
+bool SegmentIterator::_enable_ann_topn_predicate_prefilter() const {
+ return _opts.runtime_state != nullptr &&
+
_opts.runtime_state->query_options().__isset.enable_ann_topn_predicate_prefilter
&&
+
_opts.runtime_state->query_options().enable_ann_topn_predicate_prefilter;
+}
+
+Status SegmentIterator::_eager_filter_predicates_into_bitmap() {
+ // The residual column predicates (those not resolvable by
zonemap/inverted/bitmap index, hence
+ // not yet reflected in _row_bitmap) are evaluated here over the candidate
rows, and the
+ // survivors are intersected back into _row_bitmap. The narrowed bitmap is
then fed to the ANN
+ // index as an IDSelector (see
AnnTopNRuntime::evaluate_vector_ann_search), so a predicated
+ // TopN query keeps using the index instead of degrading to an O(N)
brute-force distance scan.
+ //
+ // This runs before the main scan loop sets up _range_iter / _block_rowids
/
+ // _current_columns (see _lazy_init), so it allocates the predicate
columns and drives a
+ // local pass with the same _read_columns_by_index + predicate-evaluation
primitives the main
+ // loop uses. All of those members are re-initialized by the main loop
afterwards (_range_iter
+ // is recreated over the narrowed _row_bitmap right after
_apply_ann_topn_predicate returns).
+ if (!_is_need_vec_eval && !_is_need_short_eval) {
+ // has_column_predicate was true but every predicate was already
resolved via index;
+ // there is nothing left to evaluate per row.
+ return Status::OK();
+ }
+ if (_row_bitmap.isEmpty()) {
+ return Status::OK();
+ }
+
+ const auto nrows_reserve_limit =
+ std::min(_row_bitmap.cardinality(), uint64_t(_opts.block_row_max));
+ _current_columns.resize(_schema->num_read_columns());
+ for (auto cid : _predicate_ordinals) {
+ auto storage_column_type = _storage_name_and_type[cid].second;
+ RETURN_IF_CATCH_EXCEPTION(_current_columns[cid] =
ReadSchema::get_predicate_column_ptr(
+ storage_column_type,
_opts.io_ctx.reader_type));
+ _current_columns[cid]->set_rowset_segment_id({_segment->rowset_id(),
_segment->id()});
+ _current_columns[cid]->reserve(nrows_reserve_limit);
+ }
+
+ if (_block_rowids.size() < _opts.block_row_max) {
+ _block_rowids.resize(_opts.block_row_max);
+ }
+
+ _range_iter.reset(new BitmapRangeIterator(_row_bitmap));
+ roaring::Roaring survivors;
+ while (true) {
+ // The predicate columns are reused across batches, so clear them
before each read just
+ // like the main scan loop does in _init_current_block. Without this,
_read_columns_by_index
+ // appends onto the previous batch's data and predicate evaluation
reads stale rows once the
+ // candidate set spans more than one block_row_max batch.
+ for (auto cid : _predicate_ordinals) {
+ _current_columns[cid]->clear();
+ }
+ _selected_size = 0;
+ RETURN_IF_ERROR(
+ _read_columns_by_index(_predicate_ordinals,
_opts.block_row_max, _selected_size));
Review Comment:
[P2] Account for rows read during eager predicate filtering
`_read_columns_by_index` does not update `raw_rows_read`; the normal scan
increments that counter explicitly after reading, but this eager pass never
does. With 10 million residual-predicate candidates and `LIMIT 5` (enough
survivors to use ANN), all 10 million predicate values are scanned, then only
the five ANN-selected rows reach the normal counter. `ScanRows`/audit metrics
therefore omit almost all work, and `OlapScanner::update_realtime_counters`
also forwards this undercount to `IOContext::scan_rows`, which drives
`BE_SCAN_ROWS` workload policies. Account for each eager input batch in the
scan statistics and cover the N-input/K-output case.
##########
be/src/storage/segment/segment_iterator.cpp:
##########
@@ -1350,6 +1363,83 @@ Status
SegmentIterator::_update_topn_runtime_filter_ranges() {
return Status::OK();
}
+bool SegmentIterator::_enable_ann_topn_predicate_prefilter() const {
+ return _opts.runtime_state != nullptr &&
+
_opts.runtime_state->query_options().__isset.enable_ann_topn_predicate_prefilter
&&
+
_opts.runtime_state->query_options().enable_ann_topn_predicate_prefilter;
+}
+
+Status SegmentIterator::_eager_filter_predicates_into_bitmap() {
+ // The residual column predicates (those not resolvable by
zonemap/inverted/bitmap index, hence
+ // not yet reflected in _row_bitmap) are evaluated here over the candidate
rows, and the
+ // survivors are intersected back into _row_bitmap. The narrowed bitmap is
then fed to the ANN
+ // index as an IDSelector (see
AnnTopNRuntime::evaluate_vector_ann_search), so a predicated
+ // TopN query keeps using the index instead of degrading to an O(N)
brute-force distance scan.
+ //
+ // This runs before the main scan loop sets up _range_iter / _block_rowids
/
+ // _current_columns (see _lazy_init), so it allocates the predicate
columns and drives a
+ // local pass with the same _read_columns_by_index + predicate-evaluation
primitives the main
+ // loop uses. All of those members are re-initialized by the main loop
afterwards (_range_iter
+ // is recreated over the narrowed _row_bitmap right after
_apply_ann_topn_predicate returns).
+ if (!_is_need_vec_eval && !_is_need_short_eval) {
+ // has_column_predicate was true but every predicate was already
resolved via index;
+ // there is nothing left to evaluate per row.
+ return Status::OK();
+ }
+ if (_row_bitmap.isEmpty()) {
+ return Status::OK();
+ }
+
+ const auto nrows_reserve_limit =
+ std::min(_row_bitmap.cardinality(), uint64_t(_opts.block_row_max));
+ _current_columns.resize(_schema->num_read_columns());
+ for (auto cid : _predicate_ordinals) {
+ auto storage_column_type = _storage_name_and_type[cid].second;
+ RETURN_IF_CATCH_EXCEPTION(_current_columns[cid] =
ReadSchema::get_predicate_column_ptr(
+ storage_column_type,
_opts.io_ctx.reader_type));
+ _current_columns[cid]->set_rowset_segment_id({_segment->rowset_id(),
_segment->id()});
+ _current_columns[cid]->reserve(nrows_reserve_limit);
+ }
+
+ if (_block_rowids.size() < _opts.block_row_max) {
+ _block_rowids.resize(_opts.block_row_max);
+ }
+
+ _range_iter.reset(new BitmapRangeIterator(_row_bitmap));
+ roaring::Roaring survivors;
+ while (true) {
Review Comment:
[P2] Check cancellation between eager prefilter batches
This loop now scans every candidate inside one `SegmentIterator::next_batch`
call. For a large local segment with a residual category predicate, a timeout
or `KILL QUERY` during this pass still leaves every remaining batch to be
read/evaluated, and can then load/search ANN: `BetaRowsetReader::_next_batch`
and `Scanner::get_block` only check cancellation after the batch returns, while
the scalar/page-cache/local-file read path here does not check it. The previous
fallback returned to those checks after each scan batch. Check the runtime
state's cancellation status at each eager batch boundary and return its
cancellation reason before doing more work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
