sunchao commented on PR #5612: URL: https://github.com/apache/datafusion-comet/pull/5612#issuecomment-5508929795
@dwsmith1983 Thanks for covering the requested matrix. Could you attach the runnable harness/commands, exact baseline and PR commit SHAs, dependency/build settings, and per-cell results for both replicates, including the regressing shared-instance cases? Per-task plan ownership does not rule out sharing within a task. Source inspection shows that Comet passes sort-key expressions directly to `SortExec`. In DataFusion 54.1.0, `ExternalSorter::in_mem_sort_stream` uses `spawn_buffered` for multiple retained batches once the reservation reaches `sort_in_place_threshold_bytes`. The cloned orderings retain the same expression/UDF, which can then be evaluated concurrently on Comet's multithread runtime. Could you add a native sort case with `regexp_extract_all` directly in the sort key, no LIMIT, 8192-row batches, and enough unsorted input to reach that branch? A one-/eight-worker comparison, with the native plan and evidence of overlapping calls to the same UDF, would test whether the adverse control matters here. A precomputed regex column would not exercise that sharing. This is source evidence for the path, not a reproduced end-to-end slowdown. Could you also revisit the scratch-pool attribution? `PatternCache::get_or_compile` returns an owned `Regex` clone, and the pinned regex-automata 0.4.16 `Regex::clone` creates a fresh scratch-cache pool. Sharing the compiled program is not sharing that scratch pool. The reported slowdown may still be real, but its cause needs the harness or profiling evidence. I have not independently rerun these timings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
