[
https://issues.apache.org/jira/browse/SPARK-60960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-60960:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Keep executionInfo per DataFrame, and the operation id of every ExecutePlan
> error, in Spark Connect Rust as PySpark does
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60960
> URL: https://issues.apache.org/jira/browse/SPARK-60960
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{DataFrame.executionInfo}} is the information of the
> last execution of that DataFrame, and {{None}} before one, and every error of
> an {{ExecutePlan}} request carries the request's operation id. The
> Rust-backed Python client differs:
> * {{a = spark.range(3); a.collect(); b = spark.range(5); b.executionInfo}}
> gives the information of {{a}}'s execution; PySpark gives {{None}}. The
> client keeps the last execution of the session, so a DataFrame shows another
> one's metrics and operation id, also across threads.
> * {{spark.catalog.setCurrentDatabase("nope")}} raises an
> {{AnalysisException}} whose {{operation_id}} is {{None}}; PySpark gives the
> request's operation id. The catalog operations and {{writeStream.start()}}
> send their {{ExecutePlan}} requests without the operation id of the
> reattachable path.
> Keep the execution information per DataFrame, and the operation id of every
> ExecutePlan error, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
