Hyukjin Kwon created SPARK-61329:
------------------------------------

             Summary: Draw a new seed for each call of the random functions of 
the Rust API, as PySpark does
                 Key: SPARK-61329
                 URL: https://issues.apache.org/jira/browse/SPARK-61329
             Project: Spark
          Issue Type: Bug
          Components: Connect
    Affects Versions: connect-rust-4.3.0
            Reporter: Hyukjin Kwon


The random functions of the Rust API (spark_connect::functions::rand, randn, 
uuid, shuffle, uniform, randstr and count_min_sketch) send the same fixed seed 
literal in every call, in every query, session and process. So uuid() returns 
the same UUIDs in every run (surrogate keys collide across appends), two rand() 
columns of one select are equal, and orderBy(rand()) or filter(rand() < p) pick 
the same rows every time.
PySpark 4.3.0 (pyspark.sql.connect.functions.builtin) draws a new seed with 
random.randint(0, sys.maxsize) for each call without one, and accepts an 
explicit seed; the Scala client does the same with random.nextLong. The Python 
drop-in is not affected: its functions draw from Python's random.
The fix draws a new non-negative seed for each call, adds _with_seed variants 
for an explicit seed (Long for rand and randn, a Column for the others, as the 
Scala client takes them), and adds random and random_with_seed, PySpark's alias 
of rand.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to