[
https://issues.apache.org/jira/browse/SPARK-61031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61031.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 236
[https://github.com/apache/spark-connect-rust/pull/236]
> Take the arguments of the readers, the writer and the catalog, and create the
> tables of createTable on the call, in the Python client as PySpark does
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61031
> URL: https://issues.apache.org/jira/browse/SPARK-61031
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{spark.read.load(path)}} and the readers of formats, such as
> {{csv}}, {{json}}, {{parquet}}, {{text}}, {{orc}} and {{xml}}, take a path or
> a list of paths, and {{parquet(*paths)}} takes any number of them. {{csv}},
> {{json}} and {{xml}} also take a DataFrame of one string column, which they
> parse with the schema and the options of the reader ({{Parse}}). {{json}} and
> {{xml}} raise {{NOT_EXPECTED_TYPE}} for any other {{path}}.
> {{df.write.bucketBy(numBuckets, col, *cols)}} and {{sortBy(col, *cols)}} take
> the columns as names or as a list or tuple of them. They raise
> {{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is not an int and for columns
> that are not strings, and {{CANNOT_SET_TOGETHER}} for a list together with
> more columns. {{clusterBy}} takes a list or tuple of columns too, and
> {{df.write.mode(None)}} keeps the default mode.
> {{spark.catalog.createTable}} creates the table on the call, with the schema
> and the options given. It raises {{NOT_EXPECTED_TYPE}} for a schema that is
> not a StructType. {{createExternalTable}} warns that it is deprecated and
> creates the table as {{createTable}} does. {{tableExists}},
> {{functionExists}} and {{listColumns}} warn that {{dbName}} is deprecated.
> The Rust-backed Python client raises a TypeError for a list of paths, as
> {{spark.read.csv([path])}}, and for a DataFrame of strings. It raises a
> ValueError for {{parquet}} of more than one path. It raises TypeErrors for
> the list of columns of {{bucketBy}}, {{sortBy}} and {{clusterBy}}, and for
> {{mode(None)}}, and TypeErrors rather than PySpark's errors for the other
> arguments of {{bucketBy}}. {{createTable}} and {{createExternalTable}} ignore
> the schema and the options, and create the table only when the DataFrame they
> return runs. {{createExternalTable}} and {{dbName}} do not warn. Take the
> arguments, and create the tables, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
