jizezhang commented on issue #19938: URL: https://github.com/apache/datafusion/issues/19938#issuecomment-3830477234
Hi @Dandandan , I am interested in this issue. If it has not yet been worked on, I would like to take a stab at it. Based on my understanding of the code, my thought is that we could create a new implementation of trait `GroupValues` https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/aggregates/group_values/mod.rs#L90 that is specific to integer types and separate from `GroupValuesPrimitive`https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/aggregates/group_values/single_group_by/primitive.rs#L80, which maintains a hash table internally. Logic for the new implementation would use direct indexing following similar implementation in `ArrayMap` https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/joins/array_map.rs#L103 for hash join. The logic for choosing specific implementations of `GroupValues` in `new_group_values` https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physic al-plan/src/aggregates/group_values/mod.rs#L134 would also be updated to take column statistics, available via `AggregateExec::partition_statistics`, https://github.com/apache/datafusion/blob/51c04754ee9be41195d7a197bb71a66ed1b11673/datafusion/physical-plan/src/aggregates/mod.rs#L1403 and determine whether to apply direct indexing or not. Please let me know whether this makes sense or not, and any suggestions/corrections. Thanks a lot! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
