andygrove opened a new issue, #255:
URL: https://github.com/apache/datafusion-spark/issues/255

   Part of #174 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `rlike(str, regexp)` returns true if the Java regular expression `regexp` 
matches anywhere in `str` (Spark calls `Pattern.matcher(str).find(0)`, so the 
match is unanchored), false otherwise. `regexp` and `regexp_like` are aliases 
registered since 3.2.0; `str RLIKE regexp` and `str REGEXP regexp` are the 
operator forms. `rlike` has been available since 1.0.0.
   
   Both arguments are implicitly cast to STRING (`ImplicitCastInputTypes`). 
Result is BOOLEAN; NULL if either argument is NULL (`nullIntolerant`). There is 
no flags argument. In 4.0+ the inputs are `StringTypeBinaryLcase` / 
`StringTypeWithCollation`: only `UTF8_BINARY` and `UTF8_LCASE` collations are 
accepted, and `UTF8_LCASE` compiles the pattern with `CASE_INSENSITIVE | 
UNICODE_CASE`.
   
   ### Examples
   
   ```sql
   SELECT regexp_like('1a 2b 14m', '\\d+b');
   -- true
   SELECT regexp('1a 2b 14m', '[a-z]+b');
   -- false
   SELECT rlike('%SystemDrive%\\Users\\John', '%SystemDrive%\\\\Users.*');
   -- true
   SELECT rlike('%SystemDrive%\\Users\\John', r'%SystemDrive%\\Users.*');
   -- true
   SET spark.sql.parser.escapedStringLiterals=true;
   SELECT rlike('%SystemDrive%\Users\John', '%SystemDrive%\\Users.*');
   -- true
   SELECT rlike('abbbbc', '**');
   -- error INVALID_PARAMETER_VALUE.PATTERN
   ```
   
   ### Edge cases to test
   
   - NULL `str`, NULL `regexp`, both NULL, for literal and column patterns: all 
return NULL.
   - Empty pattern: `rlike(x, '')` is true for any non-NULL `x`, including `''`.
   - Invalid pattern: error `INVALID_PARAMETER_VALUE.PATTERN` (SQLSTATE 22023), 
message ``The value of parameter(s) `regexp` in `rlike` is invalid: 
'<pattern>'.`` (the `functionName` parameter is the alias used, e.g. `` 
`regexp_like` ``). Raised for both literal and per-row patterns.
   - Non-literal (column) patterns: Spark compiles per row. Test a column with 
mixed valid patterns, and one invalid pattern in a later row.
   - Anchoring is not implicit: `rlike('afofo', '^fo')` is false, 
`rlike('afofo', 'fo')` is true (`RegexpExpressionsSuite`). `rlike('fo\no', 
'^fo\no$')` is true.
   - `spark.sql.parser.escapedStringLiterals` (default false) only changes how 
the SQL parser unescapes the literal before the function sees it; the kernel 
receives the final pattern string. Note that DataFusion's SQL parser keeps 
backslashes literal in `'...'`, so Spark `'\\d+'` is DataFusion `'\d+'` in 
`.slt` files.
   - Java vs Rust `regex` divergences (each needs a test that either matches 
Spark or is rejected):
     - Lookaround, backreferences, atomic groups, possessive quantifiers: Java 
accepts, Rust fails to compile. `(?=[^ ]*[a-z])(?=[^ ]*[0-9])[^ ]+`, `(a)\1`, 
`(?>a)`, `a*+`.
     - `\d`, `\w`, `\s` are ASCII-only in Java and Unicode in Rust: 
`rlike('\u0663', '\\d')` (ARABIC-INDIC DIGIT THREE) is false in Spark.
     - `.` excludes `\r`, `\u0085`, `\u2028`, `\u2029` in Java but only `\n` in 
Rust: `rlike('\r', '^.$')` is false in Spark.
     - `$` without `(?m)` also matches before a final line terminator in Java: 
`rlike('abc\n', 'abc$')` is true in Spark, false in Rust.
     - `(?i)` is ASCII-only in Java unless `(?u)` is also set; Rust folds 
Unicode, so `rlike('\u212A', '(?i)k')` (U+212A KELVIN SIGN) is false in Spark 
and true in Rust.
     - Flag letters differ: Java `U` is `UNICODE_CHARACTER_CLASS` but Rust `U` 
swaps greediness; Java `u` is `UNICODE_CASE`, Rust `u` toggles Unicode mode; 
Java `d` (`UNIX_LINES`) is unknown to Rust.
     - `\p{Alpha}`, `\p{Lower}` etc. are ASCII POSIX classes in Java but 
Unicode properties in Rust. `[[:alpha:]]` is an ASCII class in Rust but a 
nested class of the characters `:alph` in Java.
     - Syntax only Rust accepts: `(?P<n>...)`, `\<` / `\>`, `\b{start}`, class 
operators `--` and `~~`. Syntax only Java accepts: `\Q...\E`, `\h`, `\R`, `\X`, 
`\G`, `\Z`, `\0nn` octal, `\cX`.
   - Unicode input: multi-byte and supplementary characters such as an emoji 
(U+1F600): Java's `.` matches the whole surrogate pair and Rust's `.` matches 
the whole code point, so `rlike('😀', '^.$')` should be true on both sides; add 
a test to confirm.
   - `UTF8_LCASE` collation (4.0+): `rlike('ABC' COLLATE UTF8_LCASE, 'b')` 
matches case-insensitively.
   
   ### Differences across Spark versions
   
   - 4.0: collation-aware inputs and regex flags; result otherwise unchanged.
   - 4.2: adds an `expensive` heuristic for planning (no semantic change).
   - Error condition for invalid patterns is `INVALID_PARAMETER_VALUE.PATTERN` 
in all of 3.4 through 4.2.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
 `RLike` (base class `StringRegexExpression`). Group `predicate_funcs`.
   - DataFusion core has `regexp_like(str, regexp[, flags])` 
(`datafusion/functions/src/regex/regexplike.rs`) and the `~` operator. Both use 
Rust `regex` syntax and semantics and accept a flags argument Spark does not 
have. Wrapping them unchanged would silently differ from Spark for the cases 
above.
   - Recommended approach: a shared Java-regex compatibility helper (new 
module, e.g. `crates/spark/src/function/string/regex_utils.rs`, reused by 
`regexp_replace`, `regexp_extract_all`, `regexp_count`, `regexp_substr`, 
`regexp_instr`, `split`). It parses the pattern (for example with 
`regex-syntax`'s AST parser) and accepts it only if every construct is in a 
verified-compatible subset, translating where needed (e.g. `\d` to `[0-9]`, `.` 
to `[^\n\r\u0085\u2028\u2029]`). Patterns outside the subset should return a 
distinct "unsupported pattern" error (`not_impl_err!`), not a silent result and 
not `INVALID_PARAMETER_VALUE.PATTERN`, so that callers can fall back. For 
literal patterns, do this check at planning time (`return_field_from_args` sees 
`scalar_arguments`) so engines like Comet can decline before execution.
   - Cache the compiled regex for scalar patterns; cache the last pattern for 
column patterns.
   - `regex` is not a dependency of `crates/spark` yet. The function belongs in 
the `predicate` module (currently empty), or `string` next to `like.rs` / 
`ilike.rs`.
   - Tests: add `crates/sqllogictest/test_files/predicate/rlike.slt` (cover all 
three names) and document with `#[user_doc]`, listing the supported subset.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to