andygrove opened a new issue, #261:
URL: https://github.com/apache/datafusion-spark/issues/261
Part of #174 and apache/datafusion-comet#7300.
### What the function does
`split(str, regex[, limit])` splits `str` around matches of the Java regular
expression `regex` and returns an array. `limit` (default -1) controls how many
times the regex is applied: if `limit > 0` the array has at most `limit`
elements and the last one holds the rest of the input; if `limit <= 0` the
regex is applied as many times as possible and trailing empty strings are kept
(Spark maps `limit = 0` to -1 to avoid Java's trailing-empty removal).
Available since 1.5.0.
`str` and `regex` are implicitly cast to STRING and `limit` to INT (`limit`
may be a column). Result type is `ARRAY<STRING>` with `containsNull = false`
(element type follows `str`'s collation in 4.0+). NULL if any argument is NULL.
An empty `regex` is special-cased in `UTF8String.split`: it splits into
individual characters (code points) without the trailing empty element.
### Examples
```sql
SELECT split('oneAtwoBthreeC', '[ABC]');
-- ["one","two","three",""]
SELECT split('oneAtwoBthreeC', '[ABC]', -1);
-- ["one","two","three",""]
SELECT split('oneAtwoBthreeC', '[ABC]', 2);
-- ["one","twoBthreeC"]
SELECT split('aa1cc2ee3', '[1-9]+', 2);
-- ["aa","cc2ee3"]
SELECT split('hello', '');
-- ["h","e","l","l","o"]
SELECT split('hello', '', 1);
-- ["hello"]
SELECT split('hello', '', 3);
-- ["h","e","llo"] (4.1+; ["h","e","l"] in 3.4 to 4.0)
SELECT split('', '');
-- [""]
SELECT split('abc', null);
-- NULL
```
### Edge cases to test
- NULL `str`, `regex` or `limit`, literal and column: NULL.
- `limit` of 0, negative, 1, exactly the number of pieces, and larger.
`split('aacbbcddc', 'c', 0)` is `["aa","bb","dd",""]`
(`RegexpExpressionsSuite`), same as with -1; `split('aa2bb3cc4', '[1-9]+', -2)`
is `["aa","bb","cc",""]` (`StringFunctionsSuite`).
- Empty `regex`: `split('hello', '', 5)` and `split('hello', '', 100)` are
`["h","e","l","l","o"]`; `split('', '', n)` is `[""]` for every `n`. Splitting
is per code point (UTF-8 lead byte), so a supplementary character stays whole.
- Empty `str` with a non-empty regex: `split('', ',')` is `[""]` (Java
`String.split` returns the input when nothing matches).
- Leading empties follow Java `String.split`: a positive-width match at the
start produces a leading `""` (`split(',a', ',')` is `["","a"]`), but a
zero-width match at the start does not. `split('abc', 'x*')` is
`["a","b","c",""]` in Spark, while the `regex` crate's `split` yields a leading
`""` (`["","a","b","c",""]`).
- Empty matches after a non-empty match are reported by Java but not by the
`regex` crate (its `testdata/empty.toml`: `b|` on `abc` gives 3 matches, Java
4), so splits on patterns that can match empty differ.
- Invalid regex: `split` does not wrap the error. Spark throws the raw
`java.util.regex.PatternSyntaxException` (no error condition), unlike the other
regex functions which use `INVALID_PARAMETER_VALUE.PATTERN`. Implementations
should still raise an error.
- Column `regex` and column `limit` (`SPARK-47845` test in
`StringFunctionsSuite`).
- `spark.sql.parser.escapedStringLiterals` (default false) only affects
parsing of SQL literals before the function. In `.slt`, DataFusion keeps
backslashes literal, so `split(s, '\.')` splits on a dot.
- Java vs Rust divergences to decline or test: single-character literal
patterns take Java's fast path but match the same as regex; lookaround /
backreferences / atomic / possessive (Java only); ASCII `\d` `\w` `\s` in Java
vs Unicode in Rust (`split('a\u3000b', '\\s')`, U+3000 IDEOGRAPHIC SPACE,
returns a single element in Spark and two in Rust); `.` excludes `\r`,
`\u0085`, `\u2028`, `\u2029` in Java; `$` matches before a final line
terminator in Java; `(?i)` ASCII-only in Java; inline flag letters `U` / `u` /
`d`; Java-only `\Q..\E` (commonly used to split on a literal string), `\h`,
`\R`, `\Z`; Rust-only `(?P<n>)`, `\<`, `[[:alpha:]]`.
- `UTF8_LCASE` collation (4.0+) prefixes the regex with `(?ui)`.
### Differences across Spark versions
- 4.1.0 (SPARK-49968): with an empty regex and `0 < limit < length`, the
last element now holds the remainder (`split('abcd', '', 2)` is `["a","bcd"]`).
The old behaviour (`["a","b"]`, dropping the rest) is restored by the internal
config `spark.sql.legacy.truncateForEmptyRegexSplit` (default false). 3.4 to
4.0 always truncate.
- 4.0: collation-aware input types; element type follows `str`.
### Implementation notes
- Spark:
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
`StringSplit`, which calls `UTF8String.split` / `splitLegacyTruncate`
(`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`)
and then `java.lang.String.split(regex, limit)`.
- DataFusion core has no regex split. `string_to_array` / `split_part` split
on a literal delimiter. `datafusion-spark` has neither `split` nor a regex
helper today.
- Use the shared Java-regex compatibility helper proposed for `rlike`
(verified subset, decline otherwise, decided at planning time for a literal
regex). Implement Java `String.split` semantics directly (leading empty rule,
`limit` handling, keep trailing empties) instead of using `Regex::split` /
`splitn`. A literal regex with no metacharacters can be split as a plain
substring.
- Belongs in `crates/spark/src/function/string/`. Add
`crates/sqllogictest/test_files/string/split.slt` and document with
`#[user_doc]`.
---
_This issue was generated by an LLM (Claude) from the Apache Spark source
code and tests. Examples come from Spark's `ExpressionDescription` examples,
test suites and `.sql.out` golden files, but please verify them against Spark
before relying on them._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]