[ https://issues.apache.org/jira/browse/ARROW-13627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated ARROW-13627: ----------------------------------- Labels: kernel pull-request-available query-engine (was: kernel query-engine) > [C++] ScalarAggregateOptions don't make sense (in hash aggregation) > ------------------------------------------------------------------- > > Key: ARROW-13627 > URL: https://issues.apache.org/jira/browse/ARROW-13627 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ > Reporter: Neal Richardson > Assignee: David Li > Priority: Major > Labels: kernel, pull-request-available, query-engine > Fix For: 6.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > R's aggregation functions have a {{na.rm}} argument that governs how missing > data is handled. Assume {{x <- c(1, 2, NA, 3)}}. {{sum(x, na.rm = TRUE) == > 6}} and {{sum(x, na.rm = FALSE)}} is {{NA}} because there is at least one > missing value. > The ScalarAggregateOptions have two options: skip_nulls and min_count. From > what I can tell reading the source, in the context of sum(), skip_nulls > affects whether each element of the Array is added to "count", and if count < > min_count, you get a null value returned. So to get the expected behavior > when calling "sum" on an Array, when na.rm = TRUE, we pass skip_nulls = > false, min_count = 0. When na.rm = FALSE, we pass skip_nulls = true, > min_count = length(x), the reasoning being that you return a null value > unless all values are non-null (and count == length). See > https://github.com/apache/arrow/blob/master/r/R/compute.R#L125-L130 > This doesn't really work in the query engine, though. We don't know how many > rows are in the data to set an appropriate min_count to get the expected > behavior--the dataset being queried may have filtering. And when doing hash > aggregation, each group may have a different number of rows. -- This message was sent by Atlassian Jira (v8.3.4#803005)