Hyukjin Kwon created SPARK-61853:
------------------------------------

             Summary: Send the schema and the wholetext option of the reads as 
PySpark's readers do
                 Key: SPARK-61853
                 URL: https://issues.apache.org/jira/browse/SPARK-61853
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.4.0
            Reporter: Hyukjin Kwon


The DataSource reads of the client do not send what PySpark's readers send:
- PySpark's DataFrameReader and DataStreamReader start with _schema = "", which 
their DataSource sends as the schema "" of a read with no schema 
(pyspark/sql/connect/plan.py). The Rust core omits the field. The server reads 
both as no schema.
- PySpark's text(path, wholetext=False, ...) always sets the option wholetext, 
as "false" in the batch reader and as str(False), "False", in the stream 
reader, also over an earlier option. The Rust API's text() sets none.
So the requests of the reads differ from PySpark's, and the readwriter golden 
test of the Rust crate, which claimed protobuf parity, read no golden.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to