BiteTheDDDDt opened a new pull request, #61495: URL: https://github.com/apache/doris/pull/61495
<img width="298" height="938" alt="图片" src="https://github.com/user-attachments/assets/ddd76295-09fc-4c75-b863-dba02ba983f2" /> This pull request introduces a new bucketed hash aggregation operator for the pipeline engine, refactors aggregation data variant handling to support this new operator, and adds supporting infrastructure for efficient memory usage and operator registration. The main changes include new source and sink operator implementations for bucketed aggregation, a flexible and reusable aggregation data variant base, and various supporting improvements for memory management and code organization. **Bucketed Hash Aggregation Operator Implementation:** * Added new files `bucketed_aggregation_sink_operator.h` and `bucketed_aggregation_source_operator.h` implementing the sink and source operators for bucketed hash aggregation, including local state management, per-bucket hash tables, and pipelined merge logic. [[1]](diffhunk://#diff-88275fb8748fc49904dde78e1736a232d5a76e35c5cb97507289341de5c5f048R1-R147) [[2]](diffhunk://#diff-7762af8f72b27c4f4ddbe38793536d4b9964356b5bc21f3098f9844261649d40R1-R124) * Registered the new operators in the operator factory system, enabling their use in query execution. [[1]](diffhunk://#diff-78b7e80f67ae9b2c8f37afbd041c005e144da52962b7682619ac27bd9f4ba763R30-R31) [[2]](diffhunk://#diff-78b7e80f67ae9b2c8f37afbd041c005e144da52962b7682619ac27bd9f4ba763R826) [[3]](diffhunk://#diff-78b7e80f67ae9b2c8f37afbd041c005e144da52962b7682619ac27bd9f4ba763R858) **Aggregation Data Variant Refactoring:** * Refactored aggregation data variant logic in `agg_utils.h` to introduce a parameterized `AggMethodVariantsBase` and `AggDataVariantsBase`, supporting both traditional and bucketed aggregation with different string key hash map implementations. Added `BucketedAggDataVariants` and associated types for bucketed aggregation. [[1]](diffhunk://#diff-6ddbe2b1c190d8bc27adcd8942ffc1d7f60f47a463275aa852ea633002436334L50-R63) [[2]](diffhunk://#diff-6ddbe2b1c190d8bc27adcd8942ffc1d7f60f47a463275aa852ea633002436334L69-R203) **Performance and Memory Improvements:** * Added reusable buffers for output key storage in hash table contexts to reduce per-batch heap allocations and improve performance. These changes collectively enable efficient, parallel, and memory-aware bucketed hash aggregation in the pipeline engine, improving scalability and paving the way for further aggregation optimizations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
