Hyukjin Kwon created SPARK-61329:
------------------------------------
Summary: Draw a new seed for each call of the random functions of
the Rust API, as PySpark does
Key: SPARK-61329
URL: https://issues.apache.org/jira/browse/SPARK-61329
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
The random functions of the Rust API (spark_connect::functions::rand, randn,
uuid, shuffle, uniform, randstr and count_min_sketch) send the same fixed seed
literal in every call, in every query, session and process. So uuid() returns
the same UUIDs in every run (surrogate keys collide across appends), two rand()
columns of one select are equal, and orderBy(rand()) or filter(rand() < p) pick
the same rows every time.
PySpark 4.3.0 (pyspark.sql.connect.functions.builtin) draws a new seed with
random.randint(0, sys.maxsize) for each call without one, and accepts an
explicit seed; the Scala client does the same with random.nextLong. The Python
drop-in is not affected: its functions draw from Python's random.
The fix draws a new non-negative seed for each call, adds _with_seed variants
for an explicit seed (Long for rand and randn, a Column for the others, as the
Scala client takes them), and adds random and random_with_seed, PySpark's alias
of rand.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]