[
https://issues.apache.org/jira/browse/SPARK-61117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61117:
-----------------------------------
Labels: pull-request-available (was: )
> Call the hooks of SparkSession.Hook before each execution, in the Python
> client as PySpark does
> -----------------------------------------------------------------------------------------------
>
> Key: SPARK-61117
> URL: https://issues.apache.org/jira/browse/SPARK-61117
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark 4.3.0, {{SparkSession.Builder._registerHook(factory)}} registers a
> factory of {{SparkSession.Hook}}. The session calls each factory with itself.
> Before it sends each {{ExecutePlanRequest}}, the client calls the
> {{on_execute_plan}} of each hook with the request, and sends the request that
> the hook returns, with the operation ID of the original.
> The Python client accepts {{_registerHook}} but calls neither the factories
> nor the hooks, so a hook that adds tags to a request or changes its plan has
> no effect.
> The fix is to call the factories when the session is created, and the hooks
> before each execution of the Rust transport, with the request as a message of
> the generated {{pyspark.sql.connect.proto}} (SPARK-61111).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
