andygrove opened a new issue, #260:
URL: https://github.com/apache/datafusion-spark/issues/260

   Part of #174 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `regexp_instr(str, regexp[, idx])` returns the 1-based position where the 
first match of the Java regular expression `regexp` begins in `str`, or 0 if 
there is no match. Spark computes `Matcher.find()` then `start() + 1`. 
Available since 3.4.0.
   
   `str` and `regexp` are implicitly cast to STRING and the optional third 
argument `idx` to INT (default 0). `idx` is accepted but ignored by the 
evaluation, although a NULL `idx` still makes the result NULL. Result is INT. 
NULL if any argument is NULL.
   
   ### Examples
   
   ```sql
   SELECT regexp_instr(r"\abc", r"^\\abc$");
   -- 1
   SELECT regexp_instr('[email protected]', '@[^.]*');
   -- 5
   SELECT regexp_instr('abc', 'b');
   -- 2
   SELECT regexp_instr('abc', 'x');
   -- 0
   SELECT regexp_instr('ABC', '(?-i)b');
   -- 0
   SELECT regexp_instr('1a 2b 14m', '\\d{2}(a|b|m)');
   -- 7
   SELECT regexp_instr('abc', null);
   -- NULL
   SELECT regexp_instr('abc', col0, 1) FROM VALUES(') ?') AS t(col0);
   -- error INVALID_PARAMETER_VALUE.PATTERN
   ```
   
   ### Edge cases to test
   
   - NULL `str`, NULL `regexp`, NULL `idx` (literal and column): NULL.
   - `idx` values do not change the result: `regexp_instr('abc', 'b', 5)` and 
`regexp_instr('abc', 'b', -1)` both return 2 (no group-index check is 
performed).
   - Empty pattern: matches at position 0, so the result is 1 (also for `str = 
''`).
   - The position is `Matcher.start() + 1`, a UTF-16 code unit index, not a 
code point index. `regexp_instr('😀a', 'a')` returns 3 in Spark (the emoji is a 
surrogate pair). DataFusion's `regexp_instr` counts characters and returns 2; a 
byte-offset port would return 5.
   - Invalid pattern: `INVALID_PARAMETER_VALUE.PATTERN`, message ``The value of 
parameter(s) `regexp` in `regexp_instr` is invalid: ', ('.`` 
(`RegexpExpressionsSuite`), for literal and per-row patterns.
   - Non-literal pattern columns that change per row.
   - `spark.sql.parser.escapedStringLiterals` (default false): the first 
example uses raw literals; with the config on, `'\abc'` and `'^\\abc$'` reach 
the function unchanged. In `.slt`, DataFusion keeps backslashes literal.
   - Java vs Rust divergences to decline or test: lookaround / backreferences / 
atomic / possessive (Java only); ASCII `\d` `\w` `\s` in Java vs Unicode in 
Rust (`regexp_instr('\u06631', '\\d')`, U+0663 ARABIC-INDIC DIGIT THREE, is 2 
in Spark); `.` excludes `\r`, `\u0085`, `\u2028`, `\u2029` in Java; `$` matches 
before a final line terminator in Java (`regexp_instr('ab\n', 'b$')` is 2); 
`(?i)` ASCII-only in Java; inline flag letters (`U`, `u`, `d`) and the `(?-i)` 
example; Java-only `\Q..\E`, `\h`, `\R`, `\Z`, octal; Rust-only `(?P<n>)`, 
`\<`, `[[:alpha:]]`.
   - `UTF8_LCASE` collation (4.0+) matches case-insensitively.
   
   ### Differences across Spark versions
   
   - 4.0: collation-aware input types. Otherwise none found (3.4.0 through 
4.2.0).
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
 `RegExpInStr` (extends `RegExpExtractBase`; catches `IllegalStateException` 
and returns 0).
   - DataFusion core `regexp_instr(str, regexp[, start[, N[, flags[, 
subexpr]]]])` (`datafusion/functions/src/regex/regexpinstr.rs`) returns BIGINT, 
treats the third argument as a start position, counts code points, and uses 
Rust regex syntax. It cannot be wrapped as-is.
   - Use the shared Java-regex compatibility helper proposed for `rlike` 
(verified subset, decline otherwise, decided at planning time for literal 
patterns). Convert the Rust byte offset to a UTF-16 offset (count code points, 
adding 1 extra for each code point above U+FFFF) before adding 1.
   - Belongs in `crates/spark/src/function/string/`. Add 
`crates/sqllogictest/test_files/string/regexp_instr.slt` and document with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to