HyukjinKwon opened a new pull request, #191: URL: https://github.com/apache/spark-connect-rust/pull/191
### What changes were proposed in this pull request? [SPARK-60990](https://issues.apache.org/jira/browse/SPARK-60990): `Window.orderBy` and `WindowSpec.orderBy` now order by a column name, or a Column that is not a sort order, ascending with nulls first. PySpark's `SortOrder(expr)` defaults to that, and Spark sorts ascending that way. Before, such a column was ordered ascending with nulls last. A Column that is a sort order, such as `F.col("o").desc()`, is unchanged. ### Why are the changes needed? Window functions over a column with nulls gave other results than in PySpark. For example, over the values NULL, 1 and 2 of `o`: ```python df.select("o", F.row_number().over(Window.partitionBy("g").orderBy("o"))) ``` This gives the rows (NULL, 1), (1, 2), (2, 3) in PySpark 4.3.0, and gave (1, 1), (2, 2), (NULL, 3) here. `rank`, `lag`, running aggregates and the other functions over such a window differed the same way. `DataFrame.orderBy` and `sort` were already right. ### Does this PR introduce _any_ user-facing change? Yes, window functions over a column with nulls give the same results as in PySpark 4.3.0. ### How was this patch tested? - Against a local Spark 4.3.0 server, `row_number` over windows ordered by a name, a Column, a spec ordered twice and a descending Column gives the same rows as official PySpark 4.3.0. Before this change, three of the four differed. - A new live test checks those windows. It also passes with official PySpark 4.3.0, and fails without this change. - The lib tests and every integration test, e2e included, against the local server (647 passed), the build of all tests with `wasm-udf`, the offline Python suites (676 passed, 9 skipped), and the live drop-in and Arrow suites (70 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
