adriangb opened a new issue, #25351: URL: https://github.com/apache/datafusion/issues/25351
### Is your feature request related to a problem or challenge? A regular expression from user SQL that does not compile produces a different error for each regex function. Some of these errors look like engine faults (`Arrow error: Compute error: ...`), not like errors in the query. For `regexp_count` and `regexp_instr`, the error also drops the diagnosis from the `regex` crate. Only `regexp_like` and the `~` operators check a literal pattern before execution. All output below is from `datafusion-cli` 55.1.0, built from `main` at https://github.com/apache/datafusion/commit/8bd6629db6891640ad8c2993183131de627877e2. ```sql CREATE TABLE t (s VARCHAR, p VARCHAR); INSERT INTO t VALUES ('abc', 'a(b'); ``` #### 1. `regexp_like` and `~` / `~*`: optimizer error at planning (literal pattern), `ComputeError` at execution (column pattern) ```sql SELECT regexp_like(s, 'a(b') FROM t; -- same for s ~ 'a(b', s ~* 'a(b', regexp_like(s, 'a(b', 'i') ``` ``` Optimizer rule 'simplify_expressions' failed caused by Invalid regex caused by External error: regex parse error: a(b ^ error: unclosed group ``` This is `DataFusionError::Context("Invalid regex", External(Box<regex_syntax::Error>))` from `simplify_regex_expr` ([source](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/optimizer/src/simplify_expressions/regex.rs#L100-L106)). ```sql SELECT regexp_like(s, p) FROM t; -- same for s ~ p ``` ``` Arrow error: Compute error: Regular expression did not compile: Syntax( regex parse error: a(b ^ error: unclosed group ) ``` This is `ArrowError::ComputeError` from the arrow-rs `regexp_is_match` / `regexp_is_match_scalar` kernels (arrow-string 59.3.0, `src/regexp.rs`), called from `regexplike.rs` ([source](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/functions/src/regex/regexplike.rs#L373-L392)). If all arguments are literals and the call is not rewritten to `~`, constant folding goes through `regexp_like_scalar`, which returns a third shape, `DataFusionError::Execution` ([source](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/functions/src/regex/regexplike.rs#L424-L429)): ```sql SELECT regexp_like('abc', 'a(b', 'z'); ``` ``` Execution error: Regular expression did not compile: Syntax( regex parse error: (?z)a(b ^ error: unrecognized flag ) ``` #### 2. `regexp_match`: `ComputeError` at execution ```sql SELECT regexp_match(s, 'a(b') FROM t; ``` ``` Arrow error: Compute error: Regular expression did not compile: Syntax( regex parse error: a(b ^ error: unclosed group ) ``` The error comes from the arrow-rs `regexp_match` kernel ([call site](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/functions/src/regex/regexpmatch.rs#L200-L206)). `EXPLAIN` of this query succeeds, so the literal pattern is not checked at planning. #### 3. `regexp_count` / `regexp_instr`: `ComputeError` at execution, diagnosis lost ```sql SELECT regexp_count(s, 'a(b') FROM t; -- same for regexp_instr ``` ``` Arrow error: Compute error: Regular expression did not compile: a(b ``` `compile_regex` maps every `regex::Error` to a `ComputeError` that contains only the pattern ([source](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/functions/src/regex/mod.rs#L176-L193)). An invalid flag has the same problem: `regexp_count(s, 'a', 1, 'z')` returns `Regular expression did not compile: (?z)a`, with no indication that the flag is the cause. #### 4. `regexp_replace`: `External(regex::Error)` at execution ```sql SELECT regexp_replace(s, 'a(b', 'x') FROM t; ``` ``` External error: regex parse error: a(b ^ error: unclosed group ``` ([source](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/functions/src/regex/regexpreplace.rs#L541)) #### 5. The unsupported `g` flag is also reported in three different ways ```sql SELECT regexp_like(s, 'a', 'g') FROM t; -- Error during planning: regexp_like() does not support the "global" option SELECT regexp_match(s, 'a', 'g') FROM t; -- Error during planning: regexp_match() does not support the "global" option SELECT regexp_match(s, 'a', 'gi') FROM t; -- Arrow error: Compute error: Regular expression did not compile: Syntax( ... (?gi)a ... error: unrecognized flag ) SELECT regexp_count(s, 'a', 1, 'g') FROM t; -- Arrow error: Compute error: regexp_count()/regexp_instr() does not support the global flag ``` `regexp_like` rejects any flags value that contains `g`. `regexp_match` rejects only the exact value `g` ([source](https://github.com/apache/datafusion/blob/8bd6629db6891640ad8c2993183131de627877e2/datafusion/functions/src/regex/regexpmatch.rs#L214-L239)), so `gi` reaches the kernel. The "planning" errors for `g` are raised at execution when the flags come from a column. #### Comparison with PostgreSQL PostgreSQL 16.15 returns one user error for every function in 1–4: ``` ERROR: invalid regular expression: parentheses () not balanced ``` It returns `ERROR: invalid regular expression option: "z"` for an invalid flag, and `<function>() does not support the "global" option` for `g` in `regexp_like`, `regexp_match` and `regexp_count`. All of these have SQLSTATE class 22 (data exception), for example `2201B` for the invalid pattern. #### Why this matters - A caller cannot tell an invalid pattern in the query from an engine fault by the error type. Four of the shapes above are `ArrowError::ComputeError` or `External`, which are also used for internal failures. - For `regexp_count` and `regexp_instr`, the user does not see why the pattern is invalid. - A literal pattern is checked at planning only for `regexp_like` and the `~` operators. The other functions accept the query at planning and fail at execution, after work has started. ### Describe the solution you'd like 1. **One consistent error for a pattern that does not compile.** The regex UDFs own the pattern and the flags, so they should compile the pattern (or map the kernel error) and return `DataFusionError::Execution`, or `DataFusionError::Plan` when the pattern is a literal checked at planning. The message should contain the diagnosis from the `regex` crate, for example `regexp_count(): invalid regular expression 'a(b': unclosed group`. The error should never be a stringified `ArrowError::ComputeError`. `compile_regex` should keep the `regex::Error` instead of discarding it. 2. **Planning-time validation of literal patterns in every regex UDF**, not only `~` and `regexp_like`. A shared helper can compile a literal `(pattern, flags)` pair the same way the function assembles it at execution (`(?{flags}){pattern}`, empty flags treated as no flags, `g` stripped for `regexp_replace` and rejected for the other functions) and return a plan error with the diagnosis. The `g` check should be the same in every function (`contains('g')`). 3. **Make the `simplify_regex_expr` failure a `Plan` error** with the diagnosis, instead of `Context("Invalid regex", External(..))`. The existing tests that pin `Arrow error: Compute error: Regular expression did not compile: CompiledTooBig(10485760)` in `regexp_like.slt` and `regexp_match.slt` would change with point 1. ### Describe alternatives you've considered Change the arrow-rs kernels to return a typed error. This would not fix `regexp_count`, `regexp_instr` or `regexp_replace`, which do not use the kernels, and it would not add planning-time validation. ### Additional context An empty flags string was a related problem, fixed by https://github.com/apache/datafusion/pull/25046 (https://github.com/apache/datafusion/issues/25021). All five functions now accept `''` on `main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
