mrhhsg opened a new pull request, #66455:
URL: https://github.com/apache/doris/pull/66455
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
IN-filter metadata pruning previously converted `HybridSet` values through
generic literal and `Field` materialization paths. This duplicated the IN set
for zonemap pruning and made Bloom pruning allocate owning `Field` objects for
every candidate.
This PR:
- exposes typed `HybridSet` operations for owning min/max bounds, exact
range membership, native byte matching, cloning, and digest reuse;
- keeps only owning non-null min/max `Field` values for zonemap pruning and
performs exact checks against the source typed set;
- probes file Bloom filters directly from native `HybridSet` values with
early exit, without building a `Field` vector;
- maintains fast min/max and range summaries for tinyint/smallint bitset
containers;
- preserves published runtime-filter set immutability across multi-producer
merges while avoiding a copy for a single producer;
- keeps dictionary, ORC, and Parquet pruning behavior conservative for
unsupported or mismatched types.
Empty and NULL-only IN sets, embedded-NUL strings, borrowed string storage,
deep clones, and `IN`/`IN_OR_BLOOM` merger behavior are covered by focused
tests.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- `./run-regression-test.sh --conf
output/local-regression/regression-conf-19700.groovy --run -d
query_p1/expr_zonemap -s test_expr_zonemap_pruning_p1` (1/1 suite passed)
- [x] Unit Test
- `./run-be-ut.sh --run
--filter='ExprZonemapFilterTest.*:HybridSetTest.*:RuntimeFilterMergerTest.*:ParquetBloomFilterPruningTest.*:FileScannerV2Test.ConditionCacheDigestIncludesRuntimeFilterPayload:RuntimeFilterExprSamplingTest.deep_clone_clones_impl_tree:NewOrcReaderTest.SargDirectIn*:ParquetExprTest.test_in'
-j 16` (79/79 tests passed)
- [x] Manual test
- `BUILD_TYPE=ASAN ./build.sh --be --fe --be-extension-ignore
paimon-scanner,preload-extensions -j 16`
- `./build-support/check-format.sh`
- `CLANG_TIDY_BINARY=/tmp/doris-clang-tidy-with-resource-dir
./build-support/run-clang-tidy.sh --build-dir be/ut_build_ASAN`
- `git diff --check origin/master...HEAD`
- Focused benchmark list/run for
`(GetMinMax|LegacyMaterialize|AnyMatchRaw)` (32/32 cases completed).
- `./build.sh --benchmark -j 16` compiled, linked, and installed
`benchmark_test`; later Java-extension packaging stopped at the existing Paimon
SPI check because `org.apache.paimon.s3.S3Loader` is missing from
`META-INF/services/org.apache.paimon.fs.FileIOLoader`. This is unrelated to the
BE patch.
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
