[ 
https://issues.apache.org/jira/browse/SPARK-61521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hyukjin Kwon updated SPARK-61521:
---------------------------------
    Description: The batch writer's save and parquet/json/csv/orc/text/xml, and 
DataStreamWriter.start, reject a path that is not a str (123, a pathlib.Path) 
while their arguments are extracted, before they put the mode, the options, the 
format, partitionBy or queryName on the writer, so the writer keeps none of 
them. PySpark sets them first and raises at the path, so the writer keeps them: 
after w.json(123, compression="gzip") raises, w.mode("overwrite").save(d) 
writes gzipped JSON in PySpark and snappy Parquet here. Likewise a failed 
text() leaves the batch reader without the wholetext=false and the text format 
that PySpark's _set_opts and load set before the path check raises. (A path 
that UTF-8 cannot encode already raises the UnicodeEncodeError of PySpark's 
default protobuf backend, upb, and is unchanged.)  (was: Follow-up of 
SPARK-61497. A path with a lone surrogate (as a surrogateescape-decoded path 
is) given to read.text/csv/load, format(...).load, readStream.text/load, 
write.save/text or read.table raises UnicodeEncodeError ("'utf-8' codec can't 
encode character ..."), where PySpark raises protobuf's ValueError "'<path>' 
isn't a valid unicode string and can't be encoded in UTF-8." SPARK-61497 made 
the options raise it but left the paths. Also, a failed text() leaves the batch 
reader without the wholetext=false and the text format that PySpark's _set_opts 
and load set before the path check raises.)
        Summary: Keep the settings of a reader or writer whose path is not 
valid, as PySpark does  (was: Raise protobuf's ValueError for a path that UTF-8 
cannot encode in the readers and writers of the Python client, as PySpark does)

> Keep the settings of a reader or writer whose path is not valid, as PySpark 
> does
> --------------------------------------------------------------------------------
>
>                 Key: SPARK-61521
>                 URL: https://issues.apache.org/jira/browse/SPARK-61521
>             Project: Spark
>          Issue Type: Bug
>          Components: Connect
>    Affects Versions: connect-rust-4.3.0
>            Reporter: Hyukjin Kwon
>            Priority: Major
>             Fix For: connect-rust-4.3.0
>
>
> The batch writer's save and parquet/json/csv/orc/text/xml, and 
> DataStreamWriter.start, reject a path that is not a str (123, a pathlib.Path) 
> while their arguments are extracted, before they put the mode, the options, 
> the format, partitionBy or queryName on the writer, so the writer keeps none 
> of them. PySpark sets them first and raises at the path, so the writer keeps 
> them: after w.json(123, compression="gzip") raises, 
> w.mode("overwrite").save(d) writes gzipped JSON in PySpark and snappy Parquet 
> here. Likewise a failed text() leaves the batch reader without the 
> wholetext=false and the text format that PySpark's _set_opts and load set 
> before the path check raises. (A path that UTF-8 cannot encode already raises 
> the UnicodeEncodeError of PySpark's default protobuf backend, upb, and is 
> unchanged.)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to