david-mollitor-db opened a new pull request, #58637:
URL: https://github.com/apache/spark/pull/58637
### What changes were proposed in this pull request?
`col LIKE '%'` (a pattern of only unescaped `%`) matches every non-null
value, so a filter
`WHERE col LIKE '%'` keeps exactly the non-null rows. Today Spark evaluates
it as a per-row
regex (`.*`) and pushes nothing to the data source.
This PR rewrites such patterns, **in null-rejecting predicate positions**,
to `IsNotNull(col)`:
```
Aggregate [count(1)] Aggregate [count(1)]
+- Filter (col LIKE '%') ==> +- Filter (isnotnull(col))
+- Relation t +- Relation t
```
The rewrite is implemented in `ReplaceNullWithFalseInPredicate`, which
already performs this
class of null-as-false simplification (cf. its existing
`Not(In(...))`/`Not(InSet(...))`
handling). It fires only for a literal pattern of one or more `%` with the
escape character not
being `%`; `LIKE ''`, prefixes, and `_` patterns are excluded.
`LikeAll`/`LikeAny` are out of
scope.
### Why are the changes needed?
- `IsNotNull` is far cheaper than compiling and running a regex per row.
- `IsNotNull` pushes down: Parquet/ORC skip row groups via null-count
statistics; the raw `LIKE`
regex does not.
- If `col` is non-nullable, `IsNotNull(col)` then folds to `true` and
downstream
`PruneFilters`/`NullPropagation` drop the filter entirely.
The pattern shows up in practice from dynamically-generated SQL (e.g. an
empty search box
becoming `LIKE '%'`).
**Correctness.** `col LIKE '%'` returns `true` for a non-null value and
`null` for `null`,
whereas `IsNotNull(col)` returns `false` for `null`. They are equivalent
only in null-rejecting
predicate positions, where `null` is treated as `false`.
`ReplaceNullWithFalseInPredicate`
targets exactly those plan nodes (`Filter`, `Join` condition,
`MergeIntoTable`,
`DeleteFromTable`, `UpdateTable`, `ReplaceData`, `WriteDelta`) and recurses
only through
null-preserving operators (`And`, `Or`, `If`, `CaseWhen`), **not** into
`Not` — so
`NOT (col LIKE '%')` and `col LIKE '%'` in a projection are left untouched,
where `null` vs
`false` would be observable. No collation gate is needed: `'%'` matches
every non-null value
under any collation.
### Does this PR introduce _any_ user-facing change?
No. Query results are identical; this is a performance improvement.
### How was this patch tested?
New tests in `ReplaceNullWithFalseInPredicateSuite`:
- Rewritten: `'%'`, `'%%'`, `'%%%'` -> `IsNotNull`; through `AND`/`OR`; in a
join condition.
- Not rewritten: prefix `'a%'`, empty `''`, `%` escaped (`ESCAPE '%'`),
under `NOT`, and in a
projection (the null-vs-false safety boundary).
`build/sbt 'catalyst/testOnly *ReplaceNullWithFalseInPredicateSuite'` passes
(36/36); scalastyle
clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
