pratham76 opened a new pull request, #59233: URL: https://github.com/apache/spark/pull/59233
### What changes were proposed in this pull request? `StringSplit` (the `split(str, regex, limit)` SQL expression) now caches its compiled `java.util.regex.Pattern` and only recompiles it when the regex value actually changes, instead of recompiling on every row. This mirrors the existing `lastRegex`/`pattern` caching already used by the sibling expressions `RegExpReplace` and `RegExpExtractBase` in the same file (`regexpExpressions.scala`): `StringSplit` is marked `stateful = true`, caches the raw regex `UTF8String` (`lastRegex`), the collation-rewritten regex (`lastCollationAwareRegex`), and the compiled `Pattern`, recomputing them only when the incoming regex differs from the last one seen. Both the interpreted (`nullSafeEval`) and codegen (`doGenCode`) paths are updated. To support this, `UTF8String` gains new overloads of `split` and `splitLegacyTruncate` that accept a pre-compiled `Pattern` in addition to the raw regex `UTF8String`. ### Why are the changes needed? Previously, every row evaluated by `StringSplit` recompiled the regex `Pattern` from scratch via `String.split`, even though the regex is almost always a constant literal across an entire column. This is wasted work on every row and is inconsistent with how `RegExpReplace` and `RegExpExtractBase` already handle the identical problem in the same file. ### Does this PR introduce _any_ user-facing change? No. The output of `split()` is unchanged; this is purely a performance optimization. ### How was this patch tested? Existing UTs and have added two new tests to `RegexpExpressionsSuite` ### Was this patch authored or co-authored using generative AI tooling? Yes, Claude Sonnet 4.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
