HappenLee opened a new pull request, #70559: URL: https://github.com/apache/doris/pull/70559
### Background Doris uses segment and page zone maps to exclude data ranges before reading rows. A storage IN predicate currently compares the zone's bounds with the minimum and maximum IN values. Row-level membership checks still use the full IN set. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Enhancement Problem Summary: Sparse IN lists can retain zones containing no candidate values. For example, `IN (10, 30)` retains a zone bounded by `[11, 29]` because the bounding ranges overlap. Check the existing typed `HybridSet` for a value inside the zone after the cheap bounds check. Most of the diff adds tests for pruning boundaries and conservative fallbacks. ### Details - **Before and after:** Segment and page zone maps can now reject gaps within an IN list's bounding range. Inclusive endpoints remain candidates; row-level membership checks remain unchanged. - **Configs and defaults:** Reuse `in_zonemap_point_check_threshold` (default `8192`). Sets above the threshold retain bounds-only checks. Dense-domain containers also retain bounds-only checks to avoid scanning an entire domain for each zone. - **Impact on existing users:** Sparse IN predicates may read fewer segments and pages. Eligible checks take O(N) per zone, bounded by the configured threshold, and use O(1) additional space. No workload latency benchmark has been run. - **Behavior change:** More selective metadata pruning. Negated predicates and IN sets containing NaN retain conservative handling. ### Testing - Local ASAN on the development baseline: `./run-be-ut.sh -j 48 --run --filter='InListPredicateZoneMapTest.*:HybridSetTest.*:ExprZonemapFilterTest.*'` passed all 78 tests, including 11 new tests and an exhaustive comparison against set intersection for 741 integer ranges. - Coverage includes fixed and dynamic sets, inclusive endpoints, NULL/empty sets, negative predicates, threshold boundaries, dense-domain fallback, binary strings, predicate cloning, decimals, datetime fractions, NaNs, signed zero and infinities. `SparseIntegerSet` exercises the gap that bounds-only checking retains. - On the rebased master branch: clang-format 16, build hygiene and `git diff --check` passed. The ASAN suite has not been rerun on this rebased baseline. - Local clang-tidy on the development baseline was blocked by a pre-existing unmatched `NOLINTEND` in `core/types.h`. ### Notes for Reviewers - **Scope and reuse:** Only the storage IN predicate and its unit tests change. The implementation reuses the existing `HybridSet::contains_any_in_range` interface. - **Concurrency and lifetime:** The scan reads the predicate's existing set; the change adds no mutable predicate state, ownership changes or locks. - **Parallel paths:** Both native segment and page pruning call this method. Expression zone-map filtering already uses the range helper. Parquet statistics and delete predicates are outside this change. - **Preconditions and fallbacks:** Readers already bypass unusable zone maps. Comments and tests explain the negative-predicate, dense-domain and NaN cases. - **Compatibility:** No storage-format, protocol, persistence or write-path changes; no new configuration or FE/BE variables. - **Observability:** Existing scan pruning counters expose skipped data; no new per-zone logging is introduced. ### Release note Improve segment and page pruning for sparse IN predicates. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: local ASAN passed on the development baseline; the rebased master branch needs CI validation. See Testing for the local clang-tidy limitation. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, unit tests and this description. - Human review: Pending maintainer review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
