[
https://issues.apache.org/jira/browse/SPARK-61092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61092:
---------------------------------
Description:
In PySpark Connect, these methods of a DataFrame take their arguments, and
raise their errors, as follows:
* {{sample}} and {{randomSplit}} without a seed take a random seed in the
client ({{random.randint(0, sys.maxsize)}}), so every action of a sample
returns the same rows, and the splits of {{randomSplit}}, of one seed and a
deterministic order ({{deterministic_order}} of the {{Sample}}), are disjoint
and have every row.
* {{randomSplit(weights, seed)}} raises {{VALUE_NOT_POSITIVE}} for a negative
weight ({{arg_name}} "weights") and for weights whose sum is not positive
({{arg_name}} "sum(weights)").
* {{withMetadata(columnName, metadata)}} takes a dict of any JSON values
({{json.dumps(metadata)}}), and raises {{NOT_EXPECTED_TYPE}} for a metadata
that is not a dict.
* {{withWatermark(eventTime, delayThreshold)}} raises {{NOT_EXPECTED_TYPE}} for
an empty one or one that is not a str.
* {{df.name}} is the column {{name}} for any name but a dunder one (unless
{{PYSPARK_VALIDATE_COLUMN_NAME_LEGACY}} is 1), for which a name that is not a
column raises {{ATTRIBUTE_NOT_SUPPORTED}}; {{_jseq}}, {{_jdf}}, {{_jmap}},
{{_jcols}} and {{rdd}} raise {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
The Rust-backed Python client sends no seed, so each action of
{{df.sample(0.5)}} returns other rows, and the splits of {{randomSplit([0.5,
0.5])}} miss and repeat rows (2003 rows of 2000). It sends negative weights to
the server ({{IllegalArgumentException}}), takes only str values of metadata,
raises plain {{TypeError}}s for the others, sends an empty event time to the
server ({{PARSE_EMPTY_STATEMENT}}), and raises an {{AttributeError}} for a name
that starts with "_". Take them as PySpark does.
was:
In PySpark Connect, these methods of a DataFrame take their arguments, and
raise their errors, as follows:
* {{withMetadata(columnName, metadata)}} takes a dict of any JSON values
({{json.dumps(metadata)}}), and raises {{NOT_EXPECTED_TYPE}} for a metadata
that is not a dict.
* {{withWatermark(eventTime, delayThreshold)}} raises {{NOT_EXPECTED_TYPE}} for
an empty one or one that is not a str.
* {{randomSplit(weights, seed)}} raises {{VALUE_NOT_POSITIVE}} for a negative
weight ({{arg_name}} "weights") and for weights whose sum is not positive
({{arg_name}} "sum(weights)").
* {{df.name}} is the column {{name}} for any name but a dunder one (unless
{{PYSPARK_VALIDATE_COLUMN_NAME_LEGACY}} is 1), for which a name that is not a
column raises {{ATTRIBUTE_NOT_SUPPORTED}}; {{_jseq}}, {{_jdf}}, {{_jmap}},
{{_jcols}} and {{rdd}} raise {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
The Rust-backed Python client takes only str values of metadata, raises plain
{{TypeError}}s for the others, sends an empty event time to the server
({{PARSE_EMPTY_STATEMENT}}), sends negative weights to the server
({{IllegalArgumentException}}), and raises an {{AttributeError}} for a name
that starts with "_". Take them as PySpark does.
Summary: Take a client seed in sample and randomSplit, and the
arguments of withMetadata, withWatermark and randomSplit and the attributes of
a DataFrame, in the Python client as PySpark does (was: Take the arguments of
withMetadata, withWatermark and randomSplit, and the attributes of a DataFrame,
in the Python client as PySpark does)
> Take a client seed in sample and randomSplit, and the arguments of
> withMetadata, withWatermark and randomSplit and the attributes of a
> DataFrame, in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61092
> URL: https://issues.apache.org/jira/browse/SPARK-61092
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, these methods of a DataFrame take their arguments, and
> raise their errors, as follows:
> * {{sample}} and {{randomSplit}} without a seed take a random seed in the
> client ({{random.randint(0, sys.maxsize)}}), so every action of a sample
> returns the same rows, and the splits of {{randomSplit}}, of one seed and a
> deterministic order ({{deterministic_order}} of the {{Sample}}), are disjoint
> and have every row.
> * {{randomSplit(weights, seed)}} raises {{VALUE_NOT_POSITIVE}} for a negative
> weight ({{arg_name}} "weights") and for weights whose sum is not positive
> ({{arg_name}} "sum(weights)").
> * {{withMetadata(columnName, metadata)}} takes a dict of any JSON values
> ({{json.dumps(metadata)}}), and raises {{NOT_EXPECTED_TYPE}} for a metadata
> that is not a dict.
> * {{withWatermark(eventTime, delayThreshold)}} raises {{NOT_EXPECTED_TYPE}}
> for an empty one or one that is not a str.
> * {{df.name}} is the column {{name}} for any name but a dunder one (unless
> {{PYSPARK_VALIDATE_COLUMN_NAME_LEGACY}} is 1), for which a name that is not a
> column raises {{ATTRIBUTE_NOT_SUPPORTED}}; {{_jseq}}, {{_jdf}}, {{_jmap}},
> {{_jcols}} and {{rdd}} raise {{JVM_ATTRIBUTE_NOT_SUPPORTED}}.
> The Rust-backed Python client sends no seed, so each action of
> {{df.sample(0.5)}} returns other rows, and the splits of {{randomSplit([0.5,
> 0.5])}} miss and repeat rows (2003 rows of 2000). It sends negative weights
> to the server ({{IllegalArgumentException}}), takes only str values of
> metadata, raises plain {{TypeError}}s for the others, sends an empty event
> time to the server ({{PARSE_EMPTY_STATEMENT}}), and raises an
> {{AttributeError}} for a name that starts with "_". Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
