andygrove opened a new issue, #261:
URL: https://github.com/apache/datafusion-spark/issues/261

   Part of #174 and apache/datafusion-comet#7300.
   
   ### What the function does
   
   `split(str, regex[, limit])` splits `str` around matches of the Java regular 
expression `regex` and returns an array. `limit` (default -1) controls how many 
times the regex is applied: if `limit > 0` the array has at most `limit` 
elements and the last one holds the rest of the input; if `limit <= 0` the 
regex is applied as many times as possible and trailing empty strings are kept 
(Spark maps `limit = 0` to -1 to avoid Java's trailing-empty removal). 
Available since 1.5.0.
   
   `str` and `regex` are implicitly cast to STRING and `limit` to INT (`limit` 
may be a column). Result type is `ARRAY<STRING>` with `containsNull = false` 
(element type follows `str`'s collation in 4.0+). NULL if any argument is NULL. 
An empty `regex` is special-cased in `UTF8String.split`: it splits into 
individual characters (code points) without the trailing empty element.
   
   ### Examples
   
   ```sql
   SELECT split('oneAtwoBthreeC', '[ABC]');
   -- ["one","two","three",""]
   SELECT split('oneAtwoBthreeC', '[ABC]', -1);
   -- ["one","two","three",""]
   SELECT split('oneAtwoBthreeC', '[ABC]', 2);
   -- ["one","twoBthreeC"]
   SELECT split('aa1cc2ee3', '[1-9]+', 2);
   -- ["aa","cc2ee3"]
   SELECT split('hello', '');
   -- ["h","e","l","l","o"]
   SELECT split('hello', '', 1);
   -- ["hello"]
   SELECT split('hello', '', 3);
   -- ["h","e","llo"]   (4.1+; ["h","e","l"] in 3.4 to 4.0)
   SELECT split('', '');
   -- [""]
   SELECT split('abc', null);
   -- NULL
   ```
   
   ### Edge cases to test
   
   - NULL `str`, `regex` or `limit`, literal and column: NULL.
   - `limit` of 0, negative, 1, exactly the number of pieces, and larger. 
`split('aacbbcddc', 'c', 0)` is `["aa","bb","dd",""]` 
(`RegexpExpressionsSuite`), same as with -1; `split('aa2bb3cc4', '[1-9]+', -2)` 
is `["aa","bb","cc",""]` (`StringFunctionsSuite`).
   - Empty `regex`: `split('hello', '', 5)` and `split('hello', '', 100)` are 
`["h","e","l","l","o"]`; `split('', '', n)` is `[""]` for every `n`. Splitting 
is per code point (UTF-8 lead byte), so a supplementary character stays whole.
   - Empty `str` with a non-empty regex: `split('', ',')` is `[""]` (Java 
`String.split` returns the input when nothing matches).
   - Leading empties follow Java `String.split`: a positive-width match at the 
start produces a leading `""` (`split(',a', ',')` is `["","a"]`), but a 
zero-width match at the start does not. `split('abc', 'x*')` is 
`["a","b","c",""]` in Spark, while the `regex` crate's `split` yields a leading 
`""` (`["","a","b","c",""]`).
   - Empty matches after a non-empty match are reported by Java but not by the 
`regex` crate (its `testdata/empty.toml`: `b|` on `abc` gives 3 matches, Java 
4), so splits on patterns that can match empty differ.
   - Invalid regex: `split` does not wrap the error. Spark throws the raw 
`java.util.regex.PatternSyntaxException` (no error condition), unlike the other 
regex functions which use `INVALID_PARAMETER_VALUE.PATTERN`. Implementations 
should still raise an error.
   - Column `regex` and column `limit` (`SPARK-47845` test in 
`StringFunctionsSuite`).
   - `spark.sql.parser.escapedStringLiterals` (default false) only affects 
parsing of SQL literals before the function. In `.slt`, DataFusion keeps 
backslashes literal, so `split(s, '\.')` splits on a dot.
   - Java vs Rust divergences to decline or test: single-character literal 
patterns take Java's fast path but match the same as regex; lookaround / 
backreferences / atomic / possessive (Java only); ASCII `\d` `\w` `\s` in Java 
vs Unicode in Rust (`split('a\u3000b', '\\s')`, U+3000 IDEOGRAPHIC SPACE, 
returns a single element in Spark and two in Rust); `.` excludes `\r`, 
`\u0085`, `\u2028`, `\u2029` in Java; `$` matches before a final line 
terminator in Java; `(?i)` ASCII-only in Java; inline flag letters `U` / `u` / 
`d`; Java-only `\Q..\E` (commonly used to split on a literal string), `\h`, 
`\R`, `\Z`; Rust-only `(?P<n>)`, `\<`, `[[:alpha:]]`.
   - `UTF8_LCASE` collation (4.0+) prefixes the regex with `(?ui)`.
   
   ### Differences across Spark versions
   
   - 4.1.0 (SPARK-49968): with an empty regex and `0 < limit < length`, the 
last element now holds the remainder (`split('abcd', '', 2)` is `["a","bcd"]`). 
The old behaviour (`["a","b"]`, dropping the rest) is restored by the internal 
config `spark.sql.legacy.truncateForEmptyRegexSplit` (default false). 3.4 to 
4.0 always truncate.
   - 4.0: collation-aware input types; element type follows `str`.
   
   ### Implementation notes
   
   - Spark: 
`sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala`,
 `StringSplit`, which calls `UTF8String.split` / `splitLegacyTruncate` 
(`common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java`) 
and then `java.lang.String.split(regex, limit)`.
   - DataFusion core has no regex split. `string_to_array` / `split_part` split 
on a literal delimiter. `datafusion-spark` has neither `split` nor a regex 
helper today.
   - Use the shared Java-regex compatibility helper proposed for `rlike` 
(verified subset, decline otherwise, decided at planning time for a literal 
regex). Implement Java `String.split` semantics directly (leading empty rule, 
`limit` handling, keep trailing empties) instead of using `Regex::split` / 
`splitn`. A literal regex with no metacharacters can be split as a plain 
substring.
   - Belongs in `crates/spark/src/function/string/`. Add 
`crates/sqllogictest/test_files/string/split.slt` and document with 
`#[user_doc]`.
   
   ---
   _This issue was generated by an LLM (Claude) from the Apache Spark source 
code and tests. Examples come from Spark's `ExpressionDescription` examples, 
test suites and `.sql.out` golden files, but please verify them against Spark 
before relying on them._


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to