dwsmith1983 opened a new pull request, #5612: URL: https://github.com/apache/datafusion-comet/pull/5612
## Which issue does this PR close? No dedicated issue. Related to #4942, whose description says the remaining `Regex::new` calls were already hoisted into statics; these three user-pattern call sites were still compiling per batch on current main. ## Rationale for this change `regexp_extract`, `regexp_extract_all`, and `split` called `Regex::new` on the user's pattern inside the per-batch evaluation path, so every 8192-row batch paid a full regex compile. `rlike` in the same crate already compiles once at plan time; these three could not take that exact shape because they are scalar functions created by name, and the pattern only arrives per invocation as a scalar argument. ## What changes are included in this PR? Each planned expression now owns a one-slot `PatternCache` (new `string_funcs/pattern_cache.rs`): compile on first use, reuse while the pattern string is unchanged, recompile if it ever differs (split's serde does not require a literal pattern, so the cache tolerates changes rather than assuming a constant). `Regex` clones share the compiled program, so handing out clones per batch is an Arc bump. Error messages are byte-identical and an invalid pattern still fails at the same phase as before. Numbers on an M-series mac: criterion `regexp_extract` goes from 862us to 705us per 8192-row batch (about 18% faster), and a small-batch run (512 rows, 5000 batches) is 2.1x faster since compile cost is amortized over fewer rows. The split bench is flat because its case uses a literal delimiter, which takes the non-regex fast path. One known unknown worth stating: the cache uses a Mutex and the benches are single-threaded, so contention under DataFusion's intra-task parallelism is unmeasured. The fast path is a lock, a string compare, and a clone, so it should be negligible, and the lock also prevents duplicate compiles on a cold cache. ## How are these changes tested? Seven new tests: three cache unit tests (compile-once, recompile-on-change, invalid pattern does not poison the slot), three multi-batch tests pinning one compile across batches per function via a test-only counter, and one pinning that an invalid split pattern still errors at evaluation. Full crate suites pass (670 spark-expr, 212 core), clippy with warnings denied and fmt are clean, and the Scala side was exercised through CometStringExpressionSuite (33 tests, includes the native split path) and CometRegExpJvmSuite (46 tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
