Hyukjin Kwon created SPARK-61405:
------------------------------------

             Summary: Draw a seed of the client for sample_by without one, take 
PySpark's strata types, and fill the columns of a map in fillna_value, in the 
Rust API as PySpark does
                 Key: SPARK-61405
                 URL: https://issues.apache.org/jira/browse/SPARK-61405
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


Three differences of the Rust API's DataFrameStatFunctions.sampleBy and fillna 
from PySpark's:

1. StatFunctions::sample_by(col, fractions, None) sends no seed. The server 
then draws one (Utils.random.nextLong) each time it plans the relation, so one 
DataFrame returns different rows in different actions: count() changes from 
call to call and disagrees with collect(), and a self-join pairs two different 
samples. PySpark draws the seed on the client (random.randint(0, sys.maxsize)), 
as the Rust API's sample and random_split do since SPARK-61329's siblings.

2. sample_by takes a stratum literal of any type, but the server matches date, 
timestamp, decimal and binary strata against no row, so such a sample is 
silently empty. PySpark takes only float, int (and bool) and str keys and 
raises DISALLOWED_TYPE_FOR_CONTAINER for others.

3. fillna_value(Value::Map(..)) fails with "[NOT_EXPECTED_TYPE] Argument 
`value` should be bool, dict, float, int or str, got dict.", which names dict 
as both accepted and rejected. PySpark's fillna takes a dict and fills its 
columns with its values, ignoring subset.

Fix: draw the seed on the client when none is given; take only the strata types 
of PySpark, with its error; fill the columns of a map of string keys as 
fillna_map does, with PySpark's errors for a key that is not a string, a value 
of another type and an empty map.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to