anishmehta24 opened a new pull request, #40571: URL: https://github.com/apache/beam/pull/40571
Fixes #40570 `drop_duplicates` and `duplicated` colocated duplicates with `groupby`, which drops NaN keys (and `dropna=False` doesn't work with multiple columns, #21014). They now add a hash of the compared columns as an index level and partition on that, so duplicates still end up together. -0.0 is normalized first since pandas treats it as a duplicate of 0.0 but it hashes differently. `Series.nunique` now honours `dropna`. Added tests to frames_test.py (they fail on master with `KeyError`) and a CHANGES.md entry. frames_test.py and pandas_doctests_test.py pass. Written with help from Claude. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
