Hyukjin Kwon created SPARK-61405:
------------------------------------
Summary: Draw a seed of the client for sample_by without one, take
PySpark's strata types, and fill the columns of a map in fillna_value, in the
Rust API as PySpark does
Key: SPARK-61405
URL: https://issues.apache.org/jira/browse/SPARK-61405
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Three differences of the Rust API's DataFrameStatFunctions.sampleBy and fillna
from PySpark's:
1. StatFunctions::sample_by(col, fractions, None) sends no seed. The server
then draws one (Utils.random.nextLong) each time it plans the relation, so one
DataFrame returns different rows in different actions: count() changes from
call to call and disagrees with collect(), and a self-join pairs two different
samples. PySpark draws the seed on the client (random.randint(0, sys.maxsize)),
as the Rust API's sample and random_split do since SPARK-61329's siblings.
2. sample_by takes a stratum literal of any type, but the server matches date,
timestamp, decimal and binary strata against no row, so such a sample is
silently empty. PySpark takes only float, int (and bool) and str keys and
raises DISALLOWED_TYPE_FOR_CONTAINER for others.
3. fillna_value(Value::Map(..)) fails with "[NOT_EXPECTED_TYPE] Argument
`value` should be bool, dict, float, int or str, got dict.", which names dict
as both accepted and rejected. PySpark's fillna takes a dict and fills its
columns with its values, ignoring subset.
Fix: draw the seed on the client when none is given; take only the strata types
of PySpark, with its error; fill the columns of a map of string keys as
fillna_map does, with PySpark's errors for a key that is not a string, a value
of another type and an empty map.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]