anishmehta24 opened a new issue, #40570:
URL: https://github.com/apache/beam/issues/40570
### What happened?
In the DataFrame API, `drop_duplicates` and `duplicated` colocate duplicates
with `groupby(subset or all columns)`. groupby drops NaN keys, so these fail as
soon as a row has a NaN in one of the compared columns:
```python
import apache_beam as beam
import numpy as np
import pandas as pd
from apache_beam.dataframe.convert import to_dataframe, to_pcollection
rows = [(1.0, 'x'), (1.0, 'x'), (np.nan, 'y'), (np.nan, 'y'), (2.0, 'z')]
print(pd.DataFrame(rows, columns=['a', 'b']).drop_duplicates(keep=False))
with beam.Pipeline() as p:
df = to_dataframe(p | beam.Create([beam.Row(a=a, b=b) for a, b in rows]))
_ = (
to_pcollection(df.drop_duplicates(keep=False), yield_elements='pandas')
| beam.Map(print))
```
pandas:
```
a b
4 2.0 z
```
Beam:
```
RuntimeError: KeyError: ('y', nan) [while running ...]
```
`groupby(dropna=False)` doesn't help here because of #21014 (multiple
grouping columns).
Two related problems in `Series.nunique`, which is built on
`drop_duplicates`:
- it ignores `dropna`, so `pd.Series([1.0, 1.0, np.nan,
2.0]).nunique(dropna=False)` is 3 in pandas and 2 in Beam
- on a categorical Series with an unused category it fails with `KeyError:
'c'` (groupby on a categorical column includes unobserved categories)
Seen on Beam 2.77.0 and on master (a9f808e4c45), pandas 2.3.3, Python 3.12.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
