Hyukjin Kwon created SPARK-61392:
------------------------------------

             Summary: Report the invalid string, not a binary value, in the 
UnicodeDecodeError of a result in the Python client
                 Key: SPARK-61392
                 URL: https://issues.apache.org/jira/browse/SPARK-61392
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


A follow-up of SPARK-61264, found in a post-merge review. When a result has a 
string that is not valid UTF-8, the drop-in scans every column for the first 
invalid value, including BinaryType columns, so SELECT X'FF' AS b, CAST(X'C3' 
AS STRING) AS s reports byte 0xff of the binary column instead of byte 0xc3 of 
the string, as PySpark does. toArrow and toPandas also fail on the Arrow 
decode, where PySpark's toArrow returns the table.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to