[
https://issues.apache.org/jira/browse/SPARK-61405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61405.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 478
[https://github.com/apache/spark-connect-rust/pull/478]
> Draw a seed of the client for sample_by without one, take PySpark's strata
> types, and fill the columns of a map in fillna_value, in the Rust API as
> PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61405
> URL: https://issues.apache.org/jira/browse/SPARK-61405
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Three differences of the Rust API's DataFrameStatFunctions.sampleBy and
> fillna from PySpark's:
> 1. StatFunctions::sample_by(col, fractions, None) sends no seed. The server
> then draws one (Utils.random.nextLong) each time it plans the relation, so
> one DataFrame returns different rows in different actions: count() changes
> from call to call and disagrees with collect(), and a self-join pairs two
> different samples. PySpark draws the seed on the client (random.randint(0,
> sys.maxsize)), as the Rust API's sample and random_split do since
> SPARK-61329's siblings.
> 2. sample_by takes a stratum literal of any type, but the server matches
> date, timestamp, decimal and binary strata against no row, so such a sample
> is silently empty. PySpark takes only float, int (and bool) and str keys and
> raises DISALLOWED_TYPE_FOR_CONTAINER for others.
> 3. fillna_value(Value::Map(..)) fails with "[NOT_EXPECTED_TYPE] Argument
> `value` should be bool, dict, float, int or str, got dict.", which names dict
> as both accepted and rejected. PySpark's fillna takes a dict and fills its
> columns with its values, ignoring subset.
> Fix: draw the seed on the client when none is given; take only the strata
> types of PySpark, with its error; fill the columns of a map of string keys as
> fillna_map does, with PySpark's errors for a key that is not a string, a
> value of another type and an empty map.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]