viirya opened a new pull request, #7179: URL: https://github.com/apache/datafusion-comet/pull/7179
## Which issue does this PR close? Closes #7178. Follows up a review comment on #6130. ## Rationale for this change The native scalar Arrow UDF path fails the task when a UDF's PySpark Python version differs from the embedded interpreter's. The embedded interpreter is the `libpython` that libcomet was linked against, so its version is fixed when libcomet is built. Spark only requires the workers' Python to match the driver's. After a Python upgrade without a libcomet rebuild, every native UDF query would fail where Spark succeeds. ## What changes are included in this PR? - `NativeBase.pythonUdfVersion()` returns the embedded interpreter's `major.minor` from `Py_GetVersion`. That function may be called before the interpreter is initialized, so planning does not start Python. It returns an empty string without the `python-udf` feature. - `CometArrowEvalPythonExec.getSupportLevel` falls back to Spark when a UDF's `pythonVer` differs, and the fallback reason names both versions. - The user guide documents the fallback. Its limitations paragraph now links #6811 (interrupting a running UDF) and #7177 (`PythonException` parity). ## How are these changes tested? A Rust test checks that the version read before initialization matches `sys.version_info` afterwards. A planning test next to the profiler check uses a UDF with `pythonVer = "2.7"` and asserts the fallback and its reason. Planning-only tests that hard-coded a Python version and expected native execution now use the embedded version. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
