Hyukjin Kwon created SPARK-61863:
------------------------------------
Summary: Let other threads and Ctrl-C interrupt the native
conversion of createDataFrame, and follow PySpark's pyarrow width errors and
the cache threshold of the Rust API
Key: SPARK-61863
URL: https://issues.apache.org/jira/browse/SPARK-61863
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
Follow-ups of SPARK-61772 and SPARK-61837 for createDataFrame:
- The native conversion of local data (rows, tuples, dicts, the pandas row
path, inference) runs with no Python checkpoint and no signal check, so with
1.2M rows another Python thread pauses 2.3 to 3.9 s and Ctrl-C waits 3.3 to 5
s; PySpark's conversion is Python, which other threads and signals interrupt
(tens of ms). SPARK-61837 added such checkpoints to the result conversion only.
- A pyarrow table whose fields differ from a struct schema raises
AXIS_LENGTH_MISMATCH of a pre-check, where PySpark raises PyArrow's ValueError
of its cast; a pandas frame of no columns under StructType([]) gives [Row(),
Row()] where PySpark raises AXIS_LENGTH_MISMATCH.
- The Rust API's create_dataframe ignores the server's
localRelationCacheThreshold below 64 KiB, unlike create_dataframe_from_arrow,
the Python binding and PySpark.
- Comments still say that the pandas helpers run before or avoid a Config
request.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]