viirya opened a new issue, #7177: URL: https://github.com/apache/datafusion-comet/issues/7177
### What is the problem the feature request solves? When a scalar `@arrow_udf` runs on Comet's native path (#6130) and raises, the error reaches PySpark as a Comet native error that carries the Python traceback. Spark's own path raises `org.apache.spark.api.python.PythonException`, and PySpark's `convert_exception` turns that into `pyspark.errors.PythonException`. Code that catches `pyspark.errors.PythonException` around a query therefore behaves differently when the native path is enabled. Changing the JVM exception class is not enough on its own. `convert_exception` maps a cause of that class only when one of the cause's stack frames is in `org.apache.spark.sql.execution.python`. The native path's error is thrown from `CometExecIterator`, so PySpark would still report an `UnknownException`. ### Describe the potential solution Carry a distinct error kind for Python UDF failures from `ArrowPythonUdfExec` through the native error conversion. Have the JVM build a `PythonException` from it, at a point that satisfies PySpark's frame check, or find another way for PySpark to recognize it. Include a PySpark test that catches `pyspark.errors.PythonException` from a failing native UDF. ### Additional context Raised in review: https://github.com/apache/datafusion-comet/pull/6130#discussion_r4357444851. The current behavior is documented in the limitations paragraph of the PyArrow UDF user guide. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
