HyukjinKwon opened a new pull request, #195: URL: https://github.com/apache/spark-connect-rust/pull/195
### What changes were proposed in this pull request? [SPARK-60993](https://issues.apache.org/jira/browse/SPARK-60993) and [SPARK-60968](https://issues.apache.org/jira/browse/SPARK-60968): the SQL functions of `pyspark.sql.functions` now send their arguments as those of PySpark Connect 4.3.0 do. Some PySpark Connect implementations only marshal their arguments into a function call, with `lit`, `_to_col` and the other helpers of `pyspark.sql.connect.functions.builtin`. `scripts/gen_fn_dispatch.py` now generates such a function with that implementation's own signature and body. The generated module defines those helpers with their PySpark Connect bodies, over the literals, column references and function calls of the Rust core. That covers 527 of the 536 generated functions, and 24 that `functions.py` wrote by hand, such as `lag`, `lead`, `window`, `sha2`, `instr` and `array`. The other generated functions, `when`, `bucket` and the other partition transforms, and `sum_distinct`, still dispatch to the Rust core. So: - An optional argument is sent when it is given (SPARK-60993). For about 100 functions, the dispatch to the Rust core sent only the leading arguments, or a default instead. - An explicit None is what PySpark makes of it (SPARK-60968): no argument, as for `round(c, None)`; a NULL, as for `first(c, None)`; the default that PySpark fills in, as for `tuple_union_double`; or an error, as for `split(s, ",", None)`. - A string is a column name where PySpark takes a Column or a name, as the field of `date_part`, and arguments of other types raise PySpark's errors, as `substr(s, 1, 5)` does. Also: - `count_distinct` and `countDistinct` send all their columns, and `listagg_distinct` and `string_agg_distinct` their delimiter, through the new `count_distinct_columns`, `listagg_distinct_with_delimiter` and `string_agg_distinct_with_delimiter` of the Rust core. - `call_function` and `call_udf` check their columns as PySpark does. The generator reads the reference PySpark with the dependencies of PySpark Connect: `PYTHONPATH=<spark>/python/lib/py4j-0.10.9.9-src.zip python scripts/gen_fn_dispatch.py --pyspark-src <spark>/python`. ### Why are the changes needed? The results were silently wrong. Against a Spark 4.3.0 server: | Call | PySpark 4.3.0 and now | Before | | --- | --- | --- | | `to_date('02/01/2024', 'dd/MM/yyyy')` | 2024-01-02 | fails with `CAST_INVALID_INPUT` | | `log(2.0, 8.0)` | 3.0 | 0.693, the `ln` of 2.0 | | `sort_array(array(2, 1, 3), asc=False)` | [3, 2, 1] | [1, 2, 3] | | `split('a,b,c', ',', 2)` | ['a', 'b,c'] | ['a', 'b', 'c'] | | `substr('a,b,c', lit(1), lit(2))` | 'a,' | 'a,b,c' | | `like('a_c', lit('a!_c'), lit('!'))` | true | false | | `unix_timestamp(timestamp_seconds(1700000000))` | 1700000000 | the current time | | `months_between('2024-03-31', '2024-02-28', roundOff=False)` | 1.096774193548387 | 1.09677419 | | `first(v, ignorenulls=True)` over NULL and 5 | 5 | NULL | | `bround(2.5, None)` | 2.0 | NULL | The formats of the other parsing functions were dropped the same way, as were the modes, paddings, IVs and AADs of `aes_encrypt` and `aes_decrypt`, the seeds of `rand`, `randn`, `shuffle` and `uniform`, the delimiters of `listagg` and `string_agg`, the sketch sizes of the `hll_`, `kll_`, `theta_` and `tuple_` functions, the extra columns of `count_distinct`, and others. ### Does this PR introduce _any_ user-facing change? Yes. The SQL functions send the arguments of PySpark 4.3.0: given optional arguments, explicit Nones and column names. They raise its errors for arguments of other types, as `substr(s, 1, 5)` and `stack(2, "a", "b")` do, where PySpark takes Columns. ### How was this patch tested? - A probe makes 1446 calls of every function of `pyspark.sql.functions`. It passes the required parameters as column names, and each optional parameter as None and as values of its default's type. It calls each `*cols` function with two names and with a list of them. It compares the expressions that the client sends to a mock Connect server with those of official PySpark 4.3.0. - Before this change, 686 calls differed. Now 90 do: - 51 by the type of a decimal literal, which [SPARK-60994](https://issues.apache.org/jira/browse/SPARK-60994) tracks; - 30 in the checks of `when`, `bucket` and the UDF decorators, which [SPARK-60995](https://issues.apache.org/jira/browse/SPARK-60995) tracks (only the mock server makes some of them differ); - 9 only in the name of the Column class in Python's "object is not callable" message. - New offline tests check the reprs or errors of 55 calls, with their arguments given, None or omitted. A rewritten one checks, through the mock server, that strings are sent as literals or column names as PySpark sends them, for variant paths, `date_part` and `convert_timezone`. Their cases agree with official PySpark 4.3.0, and 46 of the 66 fail before this change. The offline tests of `substr` and `stack` now pass Columns, as PySpark requires. - A new live test checks the values of 10 of these calls. It also passes with official PySpark 4.3.0, and fails without this change. - A new unit test of the Rust core checks the new functions. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (649 passed, and 123 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (784 passed, 9 skipped), and the live drop-in and Arrow suites (77 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
