Hyukjin Kwon created SPARK-61101:
------------------------------------
Summary: Validate the save mode of DataFrameWriter, and the
arguments of DataFrameReader, in the Python client as PySpark does
Key: SPARK-61101
URL: https://issues.apache.org/jira/browse/SPARK-61101
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, the batch reader and writer check their arguments as
follows:
* {{DataFrameWriter}} raises {{UNSUPPORTED_OPERATION}} ({{PySparkValueError}})
when a write runs with a save mode other than {{append}}, {{overwrite}},
{{error}} or {{ignore}}, case-insensitively. This covers {{save}},
{{saveAsTable}}, {{insertInto}}, {{json}}, {{parquet}}, {{csv}}, {{orc}},
{{text}}, {{xml}} and {{jdbc}}.
* {{DataFrameReader._jreader}} raises {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
* {{DataFrameReader.jdbc}} with a {{column}} asserts {{lowerBound}},
{{upperBound}} and {{numPartitions}}, for example "lowerBound can not be None
when ``column`` is specified".
** It sets the format {{jdbc}} on the reader first.
** The bounds, {{numPartitions}} and the values of {{properties}} are options
of any type, as {{DataFrameWriter.jdbc}}'s {{properties}} are.
* {{DataFrameReader.changes}} of a reader with a schema raises the
{{AnalysisException}} {{_LEGACY_ERROR_TEMP_1189}} ("User specified schema not
supported with `changes`."), as {{DataStreamReader.changes}} does.
* {{format(None)}} of {{DataFrameReader}}, {{DataFrameWriter}} and
{{DataStreamReader}} leaves the format unset, so the server's default source is
used.
* {{DataFrameReader.csv}}, {{text}} and {{orc}} of None read without a path.
The Python client instead:
* accepted any save mode: {{df.write.mode("bad").save(path)}} saved the data
with the default mode
* raised an {{AttributeError}} for {{_jreader}}
* sent a JDBC read of a column without bounds to the server
* dropped the schema of a reader in {{changes}}
* raised a {{TypeError}} for {{format(None)}}
* raised an {{AssertionError}} for {{csv(None)}}, {{text(None)}} and
{{orc(None)}}
The client keeps {{errorifexists}} as the save mode {{error}}, as PySpark
documents. PySpark Connect 4.3.0 raises {{UNSUPPORTED_OPERATION}} for it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
