Jefffrey commented on code in PR #17891:
URL: https://github.com/apache/datafusion/pull/17891#discussion_r2412997325
##########
datafusion/functions-aggregate/src/array_agg.rs:
##########
@@ -687,13 +687,16 @@ impl Accumulator for OrderSensitiveArrayAggAccumulator {
// Convert array to Scalars to sort them easily. Convert back to array
at evaluation.
let array_agg_res =
ScalarValue::convert_array_to_scalar_vec(array_agg_values)?;
- for v in array_agg_res.into_iter() {
- partition_values.push(v.into());
+ for maybe_v in array_agg_res.into_iter() {
+ if let Some(v) = maybe_v {
+ partition_values.push(v.into());
+ } else {
+ partition_values.push(vec![].into());
+ }
Review Comment:
So I think doing it like this is correct if we want to keep things as they
were; it seems it's because `array_agg_values` (0th arg of state) comes from
`evaluate()`:
https://github.com/apache/datafusion/blob/0f515dc70957ab51e7f76fdf57d4b9fb4c2692fb/datafusion/functions-aggregate/src/array_agg.rs#L745-L769
That line 750 case is where the `None` might be introduced; `agg_orderings`
comes from `evaluate_orderings()` which doesn't have an equivalent:
https://github.com/apache/datafusion/blob/0f515dc70957ab51e7f76fdf57d4b9fb4c2692fb/datafusion/functions-aggregate/src/array_agg.rs#L600-L623
- I don't think it can introduce the `None` in the same way
So we can keep it as is, or potentially experiment with filtering this
`None` from both `array_agg_values` and `agg_orderings` (if we see the `None`
in `array_agg_values` then we also filter that from `agg_orderings` even though
it'll be `Some(_)` there). I don't know the impacts of this on correctness so
I'm not particularly advocating for it, but could be interesting to see what
happens 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
