Hyukjin Kwon created SPARK-60963:
------------------------------------
Summary: Retry the cache upload of large local relations in Spark
Connect Rust as PySpark does
Key: SPARK-60963
URL: https://issues.apache.org/jira/browse/SPARK-60963
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect uploads a local relation of
{{spark.sql.session.localRelationCacheThreshold}} bytes or more as cached
artifacts within its retry policy ({{_retrying()}} around {{cache_artifacts}}),
so a transient error such as {{UNAVAILABLE}} from a reset connection during the
status check or the upload is retried. The Rust client sends the
{{ArtifactStatus}} and {{AddArtifacts}} requests of {{createDataFrame}} of a
large PyArrow table once, so such an error fails the call at once. Retry the
cache upload as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
