anishmehta24 commented on code in PR #41140:
URL: https://github.com/apache/beam/pull/41140#discussion_r4531412662
##########
sdks/python/apache_beam/dataframe/frames.py:
##########
@@ -1854,8 +1854,11 @@ def _corr_aligned(self, other, min_periods):
std_x = self.std()
std_y = other.std()
cov = self._cov_aligned(other, min_periods)
+ # Like pandas, a series with no variance has a NaN correlation.
return cov.apply(
- lambda cov, std_x, std_y: cov / (std_x * std_y), args=[std_x, std_y])
+ lambda cov, std_x, std_y: cov / (std_x * std_y)
+ if std_x * std_y else float('nan'),
+ args=[std_x, std_y])
Review Comment:
Fixed in 24c5c1811c: the variance now tracks the min and max and returns
exactly 0 when they're equal, so a constant column gives NaN in all three
cases. `test_corr_no_variance` uses 0.1 now.
##########
sdks/python/apache_beam/dataframe/frames.py:
##########
@@ -3123,8 +3133,8 @@ def cov(self, min_periods, ddof):
arg_indices = []
for col in columns:
arg_indices.append((col, col))
- std = self[col].std(ddof)
- args.append(std.apply(lambda x: x*x, 'square'))
+ x = self[col].dropna()
+ args.append(x._cov_aligned(x, min_periods, ddof))
Review Comment:
Done in 24c5c1811c. The diagonals use a new `Series._var(ddof, min_periods)`
(the code behind `var`, `Arbitrary()` partitioning), so there's no extra index
shuffle. `_cov_aligned` now returns NaN for `n <= ddof` and 0.0 for one row
with `ddof=0`; added tests for both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
