andygrove opened a new issue, #255:
URL: https://github.com/apache/datafusion-spark/issues/255
Part of #174 and apache/datafusion-comet#7300.
### What the function does
`rlike(str, regexp)` returns true if the Java regular expression `regexp`
matches anywhere in `str` (Spark calls `Pattern.matcher(str).find(0)`, so the
match is unanchored), false otherwise. `regexp` and `regexp_like` are aliases
registered since 3.2.0; `str RLIKE regexp` and `str REGEXP regexp` are the
operator forms. `rlike` has been available since 1.0.0.
Both arguments are implicitly cast to STRING (`ImplicitCastInputTypes`).
Result is BOOLEAN; NULL if either argument is NULL (`nullIntolerant`). There is
no flags argument. In 4.0+ the inputs are `StringTypeBinaryLcase` /
`StringTypeWithCollation`: only `UTF8_BINARY` and `UTF8_LCASE` collations are
accepted, and `UTF8_LCASE` compiles the pattern with `CASE_INSENSITIVE |
UNICODE_CASE`.
### Examples
```sql
SELECT regexp_like('1a 2b 14m', '\\d+b');
-- true
SELECT regexp('1a 2b 14m', '[a-z]+b');
-- false
SELECT rlike('%SystemDrive%\\Users\\John', '%SystemDrive%\\\\Users.*');
-- true
SELECT rlike('%SystemDrive%\\Users\\John', r'%SystemDrive%\\Users.*');
-- true
SET spark.sql.parser.escapedStringLiterals=true;
SELECT rlike('%SystemDrive%\Users\John', '%SystemDrive%\\Users.*');
-- true
SELECT rlike('abbbbc', '**');
-- error INVALID_PARAMETER_VALUE.PATTERN
```
### Edge cases to test
- NULL `str`, NULL `regexp`, both NULL, for literal and column patterns: all
return NULL.
- Empty pattern: `rlike(x, '')` is true for any non-NULL `x`, including `''`.
- Invalid pattern: error `INVALID_PARAMETER_VALUE.PATTERN` (SQLSTATE 22023),
message ``The value of parameter(s) `regexp` in `rlike` is invalid:
'<pattern>'.`` (the `functionName` parameter is the alias used, e.g. ``
`regexp_like` ``). Raised for both literal and per-row patterns.
- Non-literal (column) patterns: Spark compiles per row. Test a column with
mixed valid patterns, and one invalid pattern in a later row.
- Anchoring is not implicit: `rlike('afofo', '^fo')` is false,
`rlike('afofo', 'fo')` is true (`RegexpExpressionsSuite`). `rlike('fo\no',
'^fo\no$')` is true.
- `spark.sql.parser.escapedStringLiterals` (default false) only changes how
the SQL parser unescapes the literal before the function sees it; the kernel
receives the final pattern string. Note that DataFusion's SQL parser keeps
backslashes literal in `'...'`, so Spark `'\\d+'` is DataFusion `'\d+'` in
`.slt` files.
- Java vs Rust `regex` divergences (each needs a test that either matches
Spark or is rejected):
- Lookaround, backreferences, atomic groups, possessive quantifiers: Java
accepts, Rust fails to compile. `(?=[^ ]*[a-z])(?=[^ ]*[0-9])[^ ]+`, `(a)\1`,
`(?>a)`, `a*+`.
- `\d`, `\w`, `\s` are ASCII-only in Java and Unicode in Rust:
`rlike('\u0663', '\\d')` (ARABIC-INDIC DIGIT THREE) is false in Spark.
- `.` excludes `\r`, `\u0085`, `\u2028`, `\u2029` in Java but only `\n` in
Rust: `rlike('\r', '^.$')` is false in Spark.
- `$` without `(?m)` also matches before a final line terminator in Java:
`rlike('abc\n', 'abc$')` is true in Spark, false in Rust.
- `(?i)` is ASCII-only in Java unless `(?u)` is also set; Rust folds
Unicode, so `rlike('\u212A', '(?i)k')` (U+212A KELVIN SIGN) is false in Spark
and true in Rust.
- Flag letters differ: Java `U` is `UNICODE_CHARACTER_CLASS` but Rust `U`
swaps greediness; Java `u` is `UNICODE_CASE`, Rust `u` toggles Unicode mode;
Java `d` (`UNIX_LINES`) is unknown to Rust.
- `\p{Alpha}`, `\p{Lower}` etc. are ASCII POSIX classes in Java but
Unicode properties in Rust. `[[:alpha:]]` is an ASCII class in Rust but a
nested class of the characters `:alph` in Java.
- Syntax only Rust accepts: `(?P<n>...)`, `\<` / `\>`, `\b{start}`, class
operators `--` and `~~`. Syntax only Java accepts: `\Q...\E`, `\h`, `\R`, `\X`,
`\G`, `\Z`, `\0nn` octal, `\cX`.
- Unicode input: multi-byte and supplementary characters such as an emoji
(U+1F600): Java's `.` matches the whole surrogate pair and Rust's `.` matches
the whole code point, so `rlike('😀', '^.$')` should be true on both sides; add
a test to confirm.
- `UTF8_LCASE` collation (4.0+): `rlike('ABC' COLLATE UTF8_LCASE, 'b')`
matches case-insensitively.
### Differences across Spark versions
- 4.0: collation-aware inputs and regex flags; result otherwise unchanged.
- 4.2: adds an `expensive` heuristic for planning (no semantic change).
- Error condition for invalid patterns is `INVALID_PARAMETER_VALUE.PATTERN`
in all of 3.4 through 4.2.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
`RLike` (base class `StringRegexExpression`). Group `predicate_funcs`.
- DataFusion core has `regexp_like(str, regexp[, flags])`
(`datafusion/functions/src/regex/regexplike.rs`) and the `~` operator. Both use
Rust `regex` syntax and semantics and accept a flags argument Spark does not
have. Wrapping them unchanged would silently differ from Spark for the cases
above.
- Recommended approach: a shared Java-regex compatibility helper (new
module, e.g. `crates/spark/src/function/string/regex_utils.rs`, reused by
`regexp_replace`, `regexp_extract_all`, `regexp_count`, `regexp_substr`,
`regexp_instr`, `split`). It parses the pattern (for example with
`regex-syntax`'s AST parser) and accepts it only if every construct is in a
verified-compatible subset, translating where needed (e.g. `\d` to `[0-9]`, `.`
to `[^\n\r\u0085\u2028\u2029]`). Patterns outside the subset should return a
distinct "unsupported pattern" error (`not_impl_err!`), not a silent result and
not `INVALID_PARAMETER_VALUE.PATTERN`, so that callers can fall back. For
literal patterns, do this check at planning time (`return_field_from_args` sees
`scalar_arguments`) so engines like Comet can decline before execution.
- Cache the compiled regex for scalar patterns; cache the last pattern for
column patterns.
- `regex` is not a dependency of `crates/spark` yet. The function belongs in
the `predicate` module (currently empty), or `string` next to `like.rs` /
`ilike.rs`.
- Tests: add `crates/sqllogictest/test_files/predicate/rlike.slt` (cover all
three names) and document with `#[user_doc]`, listing the supported subset.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]