[
https://issues.apache.org/jira/browse/SPARK-61052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61052:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Release the GIL across the requests of the writers, the catalog and the
> streaming queries in the Python client as PySpark does
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61052
> URL: https://issues.apache.org/jira/browse/SPARK-61052
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, every request to the server releases the GIL while it
> waits, as grpcio does, so other Python threads run meanwhile: during
> {{df.write.save}}, {{saveAsTable}}, {{insertInto}} and the format writers,
> the writes of {{df.writeTo}} and {{mergeInto}}, the methods of
> {{spark.catalog}}, {{df.isEmpty}}, the {{start}} and {{toTable}} of
> {{df.writeStream}}, the methods of a {{StreamingQuery}}, such as
> {{awaitTermination}}, and of {{spark.streams}}, and the pipelines of
> {{spark}}.
> The Rust-backed Python client holds the GIL across these requests, so every
> other Python thread waits for them: {{query.awaitTermination(3)}} stops every
> other thread for 3 seconds. Release the GIL across them as PySpark does, as
> the client already does for {{collect}}, {{count}} and {{sql}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
