anishmehta24 opened a new pull request, #41322: URL: https://github.com/apache/beam/pull/41322
Fixes #41321 `_liftable_agg` and `_unliftable_agg` decided whether a groupby is categorical by looking at `self._grouping_indexes`, which only lists the grouping keys that were already index levels. When grouping by columns that list is empty, so with `observed=False` the post-aggregation ran in parallel and every partition emitted every category. They now check every level of the grouped index, as #13256 did before #13843 narrowed it, so grouping by a categorical column requires Singleton partitioning like grouping by a categorical index level already does. This means `groupby(<categorical column>)` now raises `NonParallelOperation` unless non-parallel operations are allowed, instead of returning wrong results. Making it parallel is #21827. Added `test_categorical_groupby_column` (sum, mean and median; fails on master). The apache_beam/dataframe tests and the groupby, DataFrame and Series doctests pass, and the 2M-row example from the issue now matches pandas with `allow_non_parallel_operations()`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
