HyukjinKwon opened a new pull request, #218: URL: https://github.com/apache/spark-connect-rust/pull/218
### What changes were proposed in this pull request? [SPARK-61012](https://issues.apache.org/jira/browse/SPARK-61012): `sampleBy` now sends the fraction of each stratum, and takes a column name or a Column, as PySpark Connect's does. - The `StatSampleBy` relation has the fraction of each stratum and the column as an expression. - `df.sampleBy(col, fractions, seed)` and `df.stat.sampleBy(...)` raise `NOT_EXPECTED_TYPE` for a column that is neither a name nor a Column, and for fractions that are not a dict. They raise `DISALLOWED_TYPE_FOR_CONTAINER` for a stratum that is not a float, an int or a string. In the Rust API, `StatFunctions::sample_by` takes the column as anything that converts into a Column. ### Why are the changes needed? The strata were sent without their fractions, so every fraction was 0 and the sample was empty. `df.sampleBy("k", {"a": 1.0, "b": 0.0}, seed=1)` returned no rows, where PySpark 4.3.0 returns the rows of `"a"`. ### Does this PR introduce _any_ user-facing change? Yes. `sampleBy` samples by its fractions, takes a Column, and raises PySpark's errors for arguments of other types. ### How was this patch tested? - A new live test samples by the fractions of the strata, by a column name and by a Column. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the errors of the arguments. - The plan coverage test checks the fractions of the relation. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (671 passed, and 129 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1625 passed, 8 skipped), and the live drop-in and Arrow suites (95 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
