[
https://issues.apache.org/jira/browse/SPARK-61101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61101:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Validate the save mode of DataFrameWriter, and the arguments of
> DataFrameReader, in the Python client as PySpark does
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61101
> URL: https://issues.apache.org/jira/browse/SPARK-61101
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, the batch reader and writer check their arguments as
> follows:
> * {{DataFrameWriter}} raises {{UNSUPPORTED_OPERATION}}
> ({{PySparkValueError}}) when a write runs with a save mode other than
> {{append}}, {{overwrite}}, {{error}} or {{ignore}}, case-insensitively. This
> covers {{save}}, {{saveAsTable}}, {{insertInto}}, {{json}}, {{parquet}},
> {{csv}}, {{orc}}, {{text}}, {{xml}} and {{jdbc}}.
> * {{DataFrameReader._jreader}} raises {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
> * {{DataFrameReader.jdbc}} with a {{column}} asserts {{lowerBound}},
> {{upperBound}} and {{numPartitions}}, for example "lowerBound can not be None
> when ``column`` is specified".
> ** It sets the format {{jdbc}} on the reader first.
> ** The bounds, {{numPartitions}} and the values of {{properties}} are options
> of any type, as {{DataFrameWriter.jdbc}}'s {{properties}} are.
> * {{DataFrameReader.changes}} of a reader with a schema raises the
> {{AnalysisException}} {{_LEGACY_ERROR_TEMP_1189}} ("User specified schema not
> supported with `changes`."), as {{DataStreamReader.changes}} does.
> * {{format(None)}} of {{DataFrameReader}}, {{DataFrameWriter}} and
> {{DataStreamReader}} leaves the format unset, so the server's default source
> is used.
> * {{DataFrameReader.csv}}, {{text}} and {{orc}} of None read without a path.
> The Python client instead:
> * accepted any save mode: {{df.write.mode("bad").save(path)}} saved the data
> with the default mode
> * raised an {{AttributeError}} for {{_jreader}}
> * sent a JDBC read of a column without bounds to the server
> * dropped the schema of a reader in {{changes}}
> * raised a {{TypeError}} for {{format(None)}}
> * raised an {{AssertionError}} for {{csv(None)}}, {{text(None)}} and
> {{orc(None)}}
> The client keeps {{errorifexists}} as the save mode {{error}}, as PySpark
> documents. PySpark Connect 4.3.0 raises {{UNSUPPORTED_OPERATION}} for it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
