namviet157 opened a new pull request, #58337:
URL: https://github.com/apache/spark/pull/58337
### What changes were proposed in this pull request?
This pull request fixes the tie-breaking order of pandas-on-Spark
`Series.rank(method="first", ascending=False)`.
The value column follows the requested ascending direction, while the
natural-order column is always ordered ascending so that equal values are
ranked according to their original occurrence order.
A regression test for descending ranking with `method="first"` is added.
### Why are the changes needed?
The existing implementation applies descending order to both the value
column and the natural-order tie-breaker. This reverses the original order of
equal values and produces results inconsistent with pandas.
For `pd.Series([1, 2, 3, 1])`, pandas returns:
`[3.0, 2.0, 1.0, 4.0]`
The previous pandas-on-Spark result was:
`[4.0, 2.0, 1.0, 3.0]`
### Does this PR introduce _any_ user-facing change?
Yes. `Series.rank(method="first", ascending=False)` now preserves the
original occurrence order when breaking ties, matching pandas behavior.
### How was this patch tested?
A regression assertion was added to `SeriesStatTests.test_rank`.
The test failed before the fix with a 50% value difference and passed after
the fix:
./python/run-tests --python-executables python3 \
--testnames 'pyspark.pandas.tests.series.test_stat
SeriesStatTests.test_rank' \
-p 1
The Spark assembly build, Ruff checks, Ruff formatting check, and `git diff
--check` also passed.
This contribution was prepared and reviewed by our university project team,
including Phan Trung Nhut and Bui Nam Viet. We license this contribution to the
project under the Apache License, Version 2.0.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
