david-mollitor-db commented on PR #58362:
URL: https://github.com/apache/spark/pull/58362#issuecomment-5479756794
@uros-b Thank you for this — it's an astute observation, and you were
exactly right. Under a collation-aware collation the `StartsWith`/`EndsWith`
anchors can match a code point whose UTF-8 byte length differs from the
pattern's, so the byte-length guard no longer accepts the same set of strings
as the code-point guard. Concretely under `UTF8_LCASE`: the Kelvin sign
`U+212A` is a single 3-byte code point that case-folds to `k`, so it satisfies
both `StartsWith('k')` and `EndsWith('k')` and clears `OctetLength >= 2`, yet
`LIKE 'k%k'` must be `false` (the pattern needs two `k`s).
I've updated the rewrite so the byte-length (`OctetLength`) guard is applied
only when the input collation supports binary equality
(`supportsBinaryEquality`); for other collations — of the collations `LIKE`
accepts, that's only `UTF8_LCASE` — it falls back to the original code-point
`Length` guard, which is correct there.
Perhaps more importantly, I've added unit tests to `LikeSimplificationSuite`
that evaluate the rewritten predicate against `LIKE` for both a multibyte
`UTF8_BINARY` pattern and the `UTF8_LCASE` Kelvin-sign case, so this behavior
is locked in going forward.
(Also flagging that #58374 appears to be a duplicate of this PR, proposing
the same change.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
