viirya opened a new issue, #6811: URL: https://github.com/apache/datafusion-comet/issues/6811
### What is the problem the feature request solves? The experimental native scalar Arrow UDF path from #6130 runs the Python callable on the Spark task thread, inside `tokio::task::block_in_place`. Nothing interrupts that call when Spark kills the task. Spark's own path handles this: its `MonitorThread` destroys the Python worker `spark.python.task.killTimeout` (2s by default) after the task is interrupted. With the native path, a cancelled job or the losing copy of a speculative task keeps its executor core, and keeps competing for the embedded interpreter's GIL, until the UDF returns on its own. A long-running or stuck UDF never releases them. ### Describe the potential solution Interrupt the running call when the task is killed: - Record the Python thread ID before entering the callable. - Watch the task's kill state, either through a monitor like Spark's or by having the JVM side signal native code when `TaskContext.isInterrupted()` becomes true. - Raise an exception in that thread with `PyThreadState_SetAsyncExc`, then surface the resulting Python error as the task's failure. `PyThreadState_SetAsyncExc` only takes effect when the thread next executes Python bytecode, so a call blocked inside a C extension (for example, a long PyArrow kernel that has released the GIL) is not interrupted until it returns to Python. The issue should decide how to handle that case, for example by keeping Spark's kill timeout semantics as a best effort and documenting the gap. ### Additional context The current limitation is documented in the PyArrow UDF user guide added by #6130. The concern was raised in review: https://github.com/apache/datafusion-comet/pull/6130#discussion_r4219921562 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
