Hyukjin Kwon created SPARK-61062:
------------------------------------
Summary: Upload large local relations of createDataFrame at the
cache threshold in the Rust client
Key: SPARK-61062
URL: https://issues.apache.org/jira/browse/SPARK-61062
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
createDataFrame sends every local relation inline, whatever its size, from
local rows, from the pandas row conversion and from the float64 and string
pandas fast paths, so a relation beyond the server's gRPC message size limit
(128 MiB by default) fails with an HTTP/2 error. PySpark uploads a relation at
spark.sql.session.localRelationCacheThreshold as cache artifacts. Reproduction:
spark.createDataFrame(pd.DataFrame({"v": ["x" * 10_000] * 15_000},
dtype=object), "v string").count() fails with "h2 protocol error" in the Rust
client and gives 15000 with PySpark 4.3.0.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
