Hyukjin Kwon created SPARK-61342:
------------------------------------
Summary: Raise the exception of the done call of a progress
handler from the execution, as PySpark does
Key: SPARK-61342
URL: https://issues.apache.org/jira/browse/SPARK-61342
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
When a progress handler raised on its done call (done=True), the Python drop-in
returned the result of the execution (collect, toPandas, toArrow,
toLocalIterator, count, first, show, writes, spark.sql, catalog calls,
foreachPartition) and sent the exception to sys.unraisablehook, and the Rust
API's fallible handlers' done-call error was dropped. PySpark 4.3.0 makes the
done call in Progress.__exit__, inside the with of the execution
(client/core.py _execute_and_fetch and to_table_as_iterator), so the exception
is raised from the execution, and replaces the execution's own error when there
is one.
Also, the executions that PySpark runs without a Progress (the registrations of
functions, table functions, data sources and Java functions, through _execute,
and pipeline runs, through execute_command_as_iterator) called the handlers in
the drop-in; and the drop-in's removal of a server listener before a new add,
which PySpark does not send, did too.
The fix makes the done call when the stream of the execution ends, also of an
error or before the stream starts, and raises its error from the execution; an
execution dropped before its end (an iterator closed early) reports it as
before, as PySpark's generator does. The executions that PySpark runs without a
Progress no longer call the handlers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]