Gabriel39 opened a new pull request, #65369:
URL: https://github.com/apache/doris/pull/65369
[improvement](be) Read Parquet predicate columns round by round
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: The format_v2 Parquet reader materialized all predicate
columns before evaluating row-level predicates, so later predicate columns
could not reuse rows rejected by earlier single-column predicates. This change
schedules single-column predicate conjuncts with their predicate columns,
evaluates them immediately after each predicate column is read, compacts the
already-read predicate columns and current selection, and then reads later
predicate columns through ParquetColumnReader::select() for only the surviving
rows. Multi-column conjuncts and delete conjuncts still run after the needed
predicate columns are available.
### Release note
None
### Check List (For Author)
- Test: Unit Test / Regression test / Manual test
- Unit Test: attempted `./run-be-ut.sh --run
--filter='ParquetScanSelectionTest.CompactFilterShrinksCurrentSelection:ParquetScanTest.PredicateColumnsFilterRoundByRound'`
on `[email protected]:/mnt/disk3/gabriel/Workspace/dev3/doris` with fresh
`CMAKE_BUILD_DIR=/mnt/disk3/gabriel/Workspace/dev3/doris/be/ut_build_codex_parquet_rr`;
the changed `parquet_scan.cpp` and `parquet_scan_test.cpp` objects were
generated, but the global `doris_be_test` target failed before execution on
unrelated remote dirty files `be/src/core/column/predicate_column.h`,
`be/test/core/column/predicate_column_test.cpp`, and
`be/test/exprs/bitmapfilter_predicate_test.cpp`.
- Manual test: `python3 build-support/run_clang_format.py
--clang-format-executable /usr/local/opt/llvm@16/bin/clang-format --style file
--inplace true be/src/format_v2/parquet/parquet_scan.cpp
be/src/format_v2/parquet/parquet_scan.h
be/test/format_v2/parquet/parquet_scan_test.cpp` and `git diff --check`.
- Regression test: added `test_parquet_lazy_mat_profile` coverage for v2
lazy materialization with two predicates on the same Parquet predicate column;
not executed because the requested remote BE UT target is blocked by unrelated
dirty files and no external Hive regression cluster was started.
- Behavior changed: No
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
