anishmehta24 opened a new issue, #41139:
URL: https://github.com/apache/beam/issues/41139
### What happened?
`DeferredDataFrame.cov` and `corr` don't match pandas on the diagonal, and
`corr` fails the pipeline on a column with no variance.
- `cov` computes each diagonal entry as `self[col].std(ddof)` squared.
`ddof` is passed positionally, so it lands on `axis` and the diagonal always
uses `ddof=1`, and `min_periods` isn't applied to it.
- `corr` fills the diagonal with 1.0, where pandas gives NaN for a column
with fewer than `min_periods` values or no variance.
- `Series._corr_aligned` divides by `std_x * std_y`, which raises
`ZeroDivisionError` for a constant series. pandas returns NaN.
```python
import apache_beam as beam
import pandas as pd
from apache_beam.dataframe.convert import to_dataframe, to_pcollection
pdf = pd.DataFrame({
'x': [1.0, 3.0, 4.0, 3.0, 2.0, 2.0],
'y': [5.0, 5.0, 2.0, 9.0, 0.0, 1.0],
'c': [7.0] * 6,
})
print(pdf[['x', 'y']].cov(ddof=0))
with beam.Pipeline() as p:
df = to_dataframe(p | beam.Create([beam.Row(**r) for r in
pdf.to_dict('records')]))
_ = (
to_pcollection(df[['x', 'y']].cov(ddof=0), yield_elements='pandas')
| beam.Map(print))
```
pandas:
```
x y
x 0.916667 0.333333
y 0.333333 9.222222
```
Beam (2.77.0):
```
x y
x 1.100000 0.333333
y 0.333333 11.066667
```
With the same data, `cov(min_periods=7)` gives 1.1 and 11.07 on the diagonal
(pandas: all NaN), `corr(min_periods=7)` gives 1.0 there (pandas: NaN), and
`df.corr()` or `df.x.corr(df.c)` fails with `ZeroDivisionError: float division
by zero` (pandas: NaN for `c`).
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
