David Mollitor created SPARK-59355:
--------------------------------------
Summary: Simplify LIKE patterns containing escaped wildcards
Key: SPARK-59355
URL: https://issues.apache.org/jira/browse/SPARK-59355
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h3. What
{{LikeSimplification}} currently gives up on a pattern the moment it contains
the escape character (`if (pattern.contains(escapeChar)) None`), so
escaped-literal patterns stay a full per-row regex even when they are trivially
sargable once the escapes are decoded:
{code:java}
col LIKE 'ma\%ca' ==> col = 'ma%ca'
col LIKE 'abc\%def%' ==> StartsWith(col, 'abc%def')
col LIKE '%mn\%' ==> EndsWith(col, 'mn%')
col LIKE 'abbc' ESCAPE 'b' ==> col = 'abc'
{code}
This decodes valid escape sequences and applies the rule's existing shape
simplifications (EqualTo / StartsWith / EndsWith / Contains /
StartsWith+EndsWith) to the decoded literals.
h3. Why are the changes needed?
Escaped-literal patterns become sargable: the resulting
{{{}EqualTo{}}}/{{{}StartsWith{}}}/etc. push down to data sources (e.g. Parquet
prunes on {{{}StringStartsWith{}}}) and short-circuit the per-row regex,
instead of always running the regex and pushing nothing.
h3. How it works
* *Fast path (unchanged).* A quick {{pattern.contains(escapeChar)}} scan: when
the escape character is absent, the existing five regexes run exactly as before
– the common case pays nothing for the decode logic.
* *Guard.* Otherwise, if the escape character is itself {{%}} or {{_}} (a
pathological
{{ESCAPE '%'}} / {{{}ESCAPE '_'{}}}), skip – unchanged behavior. (This guard
must follow the fast-path scan: an {{ESCAPE '%'}} pattern that contains no
{{%}} is still simplifiable via the fast path.)
* *Decode.* Otherwise, walk the pattern turning {{%}} / {{_}} / \{{}} into
literal characters, treating unescaped {{{}%{}}}/{{{}_{}}} as wildcards, and
classify the decoded shape.
h3. Correctness
* *Behavior-preserving.* A valid-escape pattern's decoded literal is exactly
the string the {{LIKE}} matches, so the resulting predicate accepts the same
rows.
* *Invalid and trailing escapes are left alone.* An escape character followed
by anything other than {{{}%{}}}, {{{}_{}}}, or itself, or a trailing escape
character, makes {{LIKE}} throw at runtime
({{{}INVALID_FORMAT.ESC_IN_THE_MIDDLE{}}} / {{{}ESC_AT_THE_END{}}}); the rule
returns None for these, so the {{Like}} is kept and throws exactly as today.
* Decoded literals may contain {{{}%{}}}/{{{}_{}}}/the escape character; that
is fine, since the
predicates compare literal strings. Collation handling is unchanged. Every
shape fully replaces the {{Like}} (no residual), so no idempotency tag is
needed.
* Out of scope: an unescaped {{_}} single-character wildcard, and
{{{}LikeAll{}}}/{{{}LikeAny{}}} shapes beyond what the shared {{simplifyLike}}
already covers.
h3. Does this PR introduce any user-facing change?
No. Query results are identical (and error cases still error); this is a
performance improvement.
h3. How was this patch tested?
{{{}LikeSimplificationSuite{}}}: updated the existing escape tests (they
previously asserted the pattern was skipped and now assert the simplified form,
including the {{{}LikeAll{}}}/{{{}LikeAny{}}} multi-pattern tests whose escaped
entries now fold in), and added tests for valid-escape simplification across
all shapes and for the still-skipped cases (invalid escape, trailing escape,
{{{}ESCAPE '%'{}}}/{{{}ESCAPE '{_}'{_}{}}}{_}, and an unescaped {{_}}).
Scalastyle clean.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
