Baymine opened a new pull request, #66184:
URL: https://github.com/apache/doris/pull/66184
### What problem does this PR solve?
Issue Number: close #66183
Related PR: N/A
Problem Summary:
Today ORC/Parquet dictionary filtering only applies when a filter conjunct
is a bare column reference compared to constants. A predicate whose value side
is a deterministic single-slot string expression — e.g. `split_by_string(col,
sep)[n] = 'x'`, `element_at(...)`, `substring_index(...)` — is evaluated once
per row even though it depends only on the column value.
This lets such predicates be evaluated once per distinct dictionary value
instead of once per row. The FE (Nereids) is the authoritative admission check:
it stamps a conjunct root with `can_dict_filter` when the predicate is an
equality/IN over a single slot, is deterministic, and preserves NULL semantics
(null-in implies null-out — the dictionary carries no NULL entry, so a NULL
source must map to a NULL result to match per-row semantics). The verdict is
carried to BE via `TExprNode.can_dict_filter`; when unset (older FE) BE falls
back to a conservative allowlist. A new session variable
`enable_dict_filter_for_expr` (default true, forwarded as the
`enable_dict_filter_for_expr` query option) gates the expression case.
### Release note
Introduce dictionary-based evaluation of deterministic single-slot string
expression predicates (e.g. split_by_string(col,sep)[n]='x') on ORC/Parquet
scans, controlled by the new session variable enable_dict_filter_for_expr
(default true). Speeds up scans that filter on a heavy string expression of a
low-cardinality dictionary-encoded column.
### Check List (For Author)
- [x] Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Please explain why:
- [ ] Behavior changed:
- [ ] Does this need documentation?
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
