Hyukjin Kwon created SPARK-61853:
------------------------------------
Summary: Send the schema and the wholetext option of the reads as
PySpark's readers do
Key: SPARK-61853
URL: https://issues.apache.org/jira/browse/SPARK-61853
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.4.0
Reporter: Hyukjin Kwon
The DataSource reads of the client do not send what PySpark's readers send:
- PySpark's DataFrameReader and DataStreamReader start with _schema = "", which
their DataSource sends as the schema "" of a read with no schema
(pyspark/sql/connect/plan.py). The Rust core omits the field. The server reads
both as no schema.
- PySpark's text(path, wholetext=False, ...) always sets the option wholetext,
as "false" in the batch reader and as str(False), "False", in the stream
reader, also over an earlier option. The Rust API's text() sets none.
So the requests of the reads differ from PySpark's, and the readwriter golden
test of the Rust crate, which claimed protobuf parity, read no golden.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]