david-mollitor-db opened a new pull request, #58648:
URL: https://github.com/apache/spark/pull/58648
### What changes were proposed in this pull request?
The cost-based optimizer's `FilterEstimation` lists `StartsWith` /
`EndsWith` / `Contains` (and
`Like`) as "not supported yet", so they fall through to a conservative
default selectivity of
**1.0** (all rows pass). This estimates `StartsWith` / `EndsWith` /
`Contains` with a string
literal operand from statistics Spark already collects:
- **`nullCount`** — these predicates are null-intolerant, so a null input
never matches; only
non-null rows can match, bounding selectivity by `1 - nullPercent`
(reusing the computation in
`evaluateNullCheck`). An all-null column ⇒ 0.
- **`maxLen`** — a value must have at least as many characters as the
operand, so when the operand
is longer than the column's `maxLen`, no row can match ⇒ 0. `maxLen` is a
code-point count
(`Max(Length(col))`), so this holds under any collation.
General `Like` selectivity remains out of scope (its common
prefix/suffix/infix forms are already
rewritten to these operators before estimation runs), and `EqualTo` is
already estimated via NDV.
### Why are the changes needed?
`WHERE s LIKE 'abc%'` is rewritten to `StartsWith(s, 'abc')`; estimating it
as passing every row
distorts join planning (order, build/probe side, broadcast eligibility).
These are exactly the
predicates the LIKE simplifications produce, so today those rewrites help
pushdown/pruning but not
the CBO's cardinality estimates. Both bounds are conservative — the estimate
is at most 1.0, so it
never under-estimates (the safe direction for join planning) and only
tightens the current
default. It introduces no new statistics and is gated by the existing
CBO/column-stats
preconditions.
### Does this PR introduce _any_ user-facing change?
No. Query results are unchanged; this only refines cost-based cardinality
estimates, and only
under `spark.sql.cbo.enabled` with column statistics present.
The only TPC-DS query using `LIKE` is q91 (`hd_buy_potential LIKE
'Unknown%'`), whose injected
stats have `nullCount = 0` and `maxLen = 10 (>= len("Unknown"))`, so the
estimate equals the prior
default of 1.0 and the plan is unchanged — no plan-stability golden changes.
### How was this patch tested?
New tests in `FilterEstimationSuite`: `StartsWith` on a nullable column
(bounded by the non-null
fraction), `EndsWith`/`Contains` parity, operand longer than `maxLen` (0
rows), all-null column
(0 rows), and a non-null column (selectivity 1.0, unchanged).
`build/sbt 'catalyst/testOnly *FilterEstimationSuite'` passes (91/91);
scalastyle clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
