HyukjinKwon opened a new pull request, #296: URL: https://github.com/apache/spark-connect-rust/pull/296
### What changes were proposed in this pull request? [SPARK-61092](https://issues.apache.org/jira/browse/SPARK-61092): `sample` and `randomSplit` now choose their seed in the client, and several methods and attributes of a DataFrame take their arguments as PySpark Connect's do. As PySpark Connect's: - `sample` and `randomSplit` without a seed choose a random seed in the client (`random.randint(0, sys.maxsize)`). - Every action of a sample returns the same rows. - All splits of `randomSplit` use one seed and an input in a deterministic order (the `deterministic_order` of their `Sample`), so they are disjoint and together hold every row. - `randomSplit` raises `VALUE_NOT_POSITIVE` for a negative weight (`arg_name` "weights") and for weights whose sum is not positive (`arg_name` "sum(weights)"). - `withMetadata` takes a dict of any JSON values, which it sends as `json.dumps(metadata)`. It raises `NOT_EXPECTED_TYPE` for metadata that is not a dict. - `withWatermark` raises `NOT_EXPECTED_TYPE` for an event time or delay that is empty or not a str. - `df.name` is the column `name` for any name except a dunder one. Unless `PYSPARK_VALIDATE_COLUMN_NAME_LEGACY` is `1`, a name is only checked against the columns when it is a dunder name. A checked name that is not a column raises `ATTRIBUTE_NOT_SUPPORTED`. - `_jseq`, `_jdf`, `_jmap` and `_jcols` raise `JVM_ATTRIBUTE_NOT_SUPPORTED`, and `rdd` raises `NOT_IMPLEMENTED`. In the core, the `Sample` plan gains `deterministic_order`, and `sample` and `random_split` choose a seed when none is given. `DataFrame::with_metadata_json` sets the metadata of a column from its JSON. ### Why are the changes needed? The client sent no seed, so the server chose a new one at each execution: - each action of `df.sample(0.5)` returned other rows - the splits of `randomSplit([0.5, 0.5])` lost and repeated rows: they had 2003, 1994 and 2060 rows of 2000 in three runs The client also: - sent negative weights to the server (`IllegalArgumentException`), and an empty event time (`PARSE_EMPTY_STATEMENT`) - accepted only str values of metadata - raised plain `TypeError`s for these arguments, and an `AttributeError` for any name starting with `_` ### Does this PR introduce _any_ user-facing change? Yes. Samples and splits are consistent, and these methods take the arguments, and raise the errors, of PySpark 4.3.0. ### How was this patch tested? - A new live test checks that two actions of a sample return the same rows, that the splits of `randomSplit` are disjoint and hold every row, and the metadata of `withMetadata` of non-str values. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the errors of `randomSplit`, `withMetadata` and `withWatermark`, and the attributes of a DataFrame, including `PYSPARK_VALIDATE_COLUMN_NAME_LEGACY`. - A new lib test checks the seed of the splits and of a sample, and the deterministic order of the splits. - On a local server, the argument errors of 64 cases of DataFrame methods, and 14 attributes, give the same results as official PySpark 4.3.0. The only exceptions are the messages of `TypeError`s and `AssertionError`s that come from protobuf or from Python internals. - The lib tests of both crates (595 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2045 passed, 12 skipped), and the live drop-in and Arrow suites (148 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
