viirya opened a new issue, #7178: URL: https://github.com/apache/datafusion-comet/issues/7178
### What is the problem the feature request solves? The native scalar Arrow UDF path (#6130) checks a UDF's PySpark Python version only when a task creates its callable. A mismatch then fails the task. The embedded interpreter is the `libpython` that libcomet was linked against, so its version is fixed when libcomet is built. Spark only requires the workers' Python to match the driver's. After a Python upgrade without a libcomet rebuild, every query that selects the native path fails, while the same query succeeds on Spark's Python workers. ### Describe the potential solution Expose the embedded interpreter's `major.minor` version to the JVM through `NativeBase`, read with `Py_GetVersion`, which may be called before the interpreter is initialized. Have `CometArrowEvalPythonExec.getSupportLevel` fall back to Spark when a UDF's `pythonVer` differs, and add a planning test. ### Additional context Raised in review: https://github.com/apache/datafusion-comet/pull/6130#discussion_r4357444630 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
