[
https://issues.apache.org/jira/browse/SPARK-61031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61031:
---------------------------------
Description:
In PySpark, {{spark.read.load(path)}} and the readers of formats, such as
{{csv}}, {{json}}, {{parquet}}, {{text}}, {{orc}} and {{xml}}, take a path or a
list of paths, and {{parquet(*paths)}} takes any number of them. {{csv}},
{{json}} and {{xml}} also take a DataFrame of one string column, which they
parse with the schema and the options of the reader ({{Parse}}). {{json}} and
{{xml}} raise {{NOT_EXPECTED_TYPE}} for any other {{path}}.
{{df.write.bucketBy(numBuckets, col, *cols)}} and {{sortBy(col, *cols)}} take
the columns as names or as a list or tuple of them. They raise
{{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is not an int and for columns
that are not strings, and {{CANNOT_SET_TOGETHER}} for a list together with more
columns. {{clusterBy}} takes a list or tuple of columns too, and
{{df.write.mode(None)}} keeps the default mode.
{{spark.catalog.createTable}} creates the table on the call, with the schema
and the options given. It raises {{NOT_EXPECTED_TYPE}} for a schema that is not
a StructType. {{createExternalTable}} warns that it is deprecated and creates
the table as {{createTable}} does. {{tableExists}}, {{functionExists}} and
{{listColumns}} warn that {{dbName}} is deprecated.
The Rust-backed Python client raises a TypeError for a list of paths, as
{{spark.read.csv([path])}}, and for a DataFrame of strings. It raises a
ValueError for {{parquet}} of more than one path. It raises TypeErrors for the
list of columns of {{bucketBy}}, {{sortBy}} and {{clusterBy}}, and for
{{mode(None)}}, and TypeErrors rather than PySpark's errors for the other
arguments of {{bucketBy}}. {{createTable}} and {{createExternalTable}} ignore
the schema and the options, and create the table only when the DataFrame they
return runs. {{createExternalTable}} and {{dbName}} do not warn. Take the
arguments, and create the tables, as PySpark does.
was:
In PySpark, {{spark.read.load(path)}} and the readers of formats, such as
{{csv}}, {{json}}, {{text}}, {{orc}} and {{xml}}, take a path or a list of
paths. {{df.write.bucketBy(numBuckets, col, *cols)}} and {{sortBy(col, *cols)}}
take the columns as names or as a list or tuple of them, and raise
{{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is not an int and for columns
that are not strings. {{df.write.mode(None)}} keeps the default mode.
{{spark.catalog.createTable}} raises {{NOT_EXPECTED_TYPE}} for a schema that is
not a StructType, and {{tableExists(tableName, dbName)}} warns that {{dbName}}
is deprecated.
The Rust-backed Python client raises a TypeError for a list of paths, as
{{spark.read.csv([path])}}, for the list of columns of {{bucketBy}} and
{{sortBy}}, and for {{mode(None)}}. It raises TypeErrors rather than PySpark's
errors for the arguments of {{bucketBy}}, sends a schema of {{createTable}}
that is not a StructType to the server, and does not warn of {{dbName}}. Take
the arguments as PySpark does.
> Take the lists of paths of the readers and of columns of bucketBy and sortBy,
> and the arguments of mode and createTable, in the Python client as PySpark
> does
> -------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61031
> URL: https://issues.apache.org/jira/browse/SPARK-61031
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark, {{spark.read.load(path)}} and the readers of formats, such as
> {{csv}}, {{json}}, {{parquet}}, {{text}}, {{orc}} and {{xml}}, take a path or
> a list of paths, and {{parquet(*paths)}} takes any number of them. {{csv}},
> {{json}} and {{xml}} also take a DataFrame of one string column, which they
> parse with the schema and the options of the reader ({{Parse}}). {{json}} and
> {{xml}} raise {{NOT_EXPECTED_TYPE}} for any other {{path}}.
> {{df.write.bucketBy(numBuckets, col, *cols)}} and {{sortBy(col, *cols)}} take
> the columns as names or as a list or tuple of them. They raise
> {{NOT_EXPECTED_TYPE}} for a {{numBuckets}} that is not an int and for columns
> that are not strings, and {{CANNOT_SET_TOGETHER}} for a list together with
> more columns. {{clusterBy}} takes a list or tuple of columns too, and
> {{df.write.mode(None)}} keeps the default mode.
> {{spark.catalog.createTable}} creates the table on the call, with the schema
> and the options given. It raises {{NOT_EXPECTED_TYPE}} for a schema that is
> not a StructType. {{createExternalTable}} warns that it is deprecated and
> creates the table as {{createTable}} does. {{tableExists}},
> {{functionExists}} and {{listColumns}} warn that {{dbName}} is deprecated.
> The Rust-backed Python client raises a TypeError for a list of paths, as
> {{spark.read.csv([path])}}, and for a DataFrame of strings. It raises a
> ValueError for {{parquet}} of more than one path. It raises TypeErrors for
> the list of columns of {{bucketBy}}, {{sortBy}} and {{clusterBy}}, and for
> {{mode(None)}}, and TypeErrors rather than PySpark's errors for the other
> arguments of {{bucketBy}}. {{createTable}} and {{createExternalTable}} ignore
> the schema and the options, and create the table only when the DataFrame they
> return runs. {{createExternalTable}} and {{dbName}} do not warn. Take the
> arguments, and create the tables, as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
