viirya opened a new pull request, #6130: URL: https://github.com/apache/datafusion-comet/pull/6130
## Which issue does this PR close? Closes #6129. ## Rationale for this change Spark 4.1 scalar `@arrow_udf` expressions currently fall back at `ArrowEvalPythonExec`, interrupting Comet's native pipeline. An opt-in in-process path keeps Arrow batches in Comet and avoids the Spark Python worker IPC path. ## What changes are included in this PR? - Add an optional `python-udf` Cargo feature and a native `ArrowPythonUdfExec` that invokes Python through PyO3 and exchanges arrays through the Arrow C Data interface. - Serialize supported Spark 4.1+ scalar Arrow UDFs into the native plan, with a capability check and clear fallback for unsupported cases. Earlier Spark versions retain their current behavior. - Add `spark.comet.exec.nativeArrowPythonUDF.enabled`, defaulting to `false`, plus user documentation, Rust tests, Spark integration tests, and an opt-in benchmark suite. - Register the integration suite in the Linux and macOS test matrices. The callable still executes in Python. This PR supports scalar Arrow UDFs only; regular Python UDFs and pandas UDFs remain outside this path. ## How are these changes tested? - Five Rust unit tests passed with `--features python-udf`. - Spark 4.1 integration tests passed with a feature-enabled native library, covering native plan selection, results, multiple functions, and fallback settings. - Spark 3.5, 4.0, 4.1, and 4.2 profiles compiled; Spark 4.2 test sources compiled. - The opt-in Spark 4.1.3 benchmark validated both physical plans and result checksums. For 50 million rows with `pyarrow.compute.negate`, five measured iterations after warmup gave a median of 1.276 s for Spark `ArrowEvalPythonExec` and 0.468 s for the native path (2.73x). With Comet aggregation and shuffle disabled, the medians were 1.341 s and 0.407 s (3.30x). These are single-machine measurements. The `run-spark-4.1-tests` and `run-pyarrow-udf-tests` labels request broader CI coverage before merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
