[
https://issues.apache.org/jira/browse/BEAM-12495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17549417#comment-17549417
]
Danny McCormick commented on BEAM-12495:
----------------------------------------
This issue has been migrated to https://github.com/apache/beam/issues/21014
> DataFrame API: groupby(dropna=False) still drops NAs when grouping on
> multiple columns or indexes
> -------------------------------------------------------------------------------------------------
>
> Key: BEAM-12495
> URL: https://issues.apache.org/jira/browse/BEAM-12495
> Project: Beam
> Issue Type: Bug
> Components: dsl-dataframe, sdk-py-core
> Reporter: Brian Hulette
> Priority: P3
> Labels: dataframe-api
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> {code}
> df.groupby(['foo', 'bar'], dropna=False).sum()
> {code}
> This will still drop NAs in the output.
> This is due to pandas bug
> [36470|https://github.com/pandas-dev/pandas/issues/36470] "BUG: groupby(...,
> dropna=False) excludes NA values when grouping on MultiIndex levels".
> We implement groupby by moving all grouped data into the index and requiring
> Index() partitioning, so we will always run into this issue, even when the
> user is grouping on columns, not indexes.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
