BiteTheDDDDt opened a new pull request, #61495:
URL: https://github.com/apache/doris/pull/61495

   
   <img width="298" height="938" alt="图片" 
src="https://github.com/user-attachments/assets/ddd76295-09fc-4c75-b863-dba02ba983f2";
 />
   
   
   
   This pull request introduces a new bucketed hash aggregation operator for 
the pipeline engine, refactors aggregation data variant handling to support 
this new operator, and adds supporting infrastructure for efficient memory 
usage and operator registration. The main changes include new source and sink 
operator implementations for bucketed aggregation, a flexible and reusable 
aggregation data variant base, and various supporting improvements for memory 
management and code organization.
   
   **Bucketed Hash Aggregation Operator Implementation:**
   
   * Added new files `bucketed_aggregation_sink_operator.h` and 
`bucketed_aggregation_source_operator.h` implementing the sink and source 
operators for bucketed hash aggregation, including local state management, 
per-bucket hash tables, and pipelined merge logic. 
[[1]](diffhunk://#diff-88275fb8748fc49904dde78e1736a232d5a76e35c5cb97507289341de5c5f048R1-R147)
 
[[2]](diffhunk://#diff-7762af8f72b27c4f4ddbe38793536d4b9964356b5bc21f3098f9844261649d40R1-R124)
   * Registered the new operators in the operator factory system, enabling 
their use in query execution. 
[[1]](diffhunk://#diff-78b7e80f67ae9b2c8f37afbd041c005e144da52962b7682619ac27bd9f4ba763R30-R31)
 
[[2]](diffhunk://#diff-78b7e80f67ae9b2c8f37afbd041c005e144da52962b7682619ac27bd9f4ba763R826)
 
[[3]](diffhunk://#diff-78b7e80f67ae9b2c8f37afbd041c005e144da52962b7682619ac27bd9f4ba763R858)
   
   **Aggregation Data Variant Refactoring:**
   
   * Refactored aggregation data variant logic in `agg_utils.h` to introduce a 
parameterized `AggMethodVariantsBase` and `AggDataVariantsBase`, supporting 
both traditional and bucketed aggregation with different string key hash map 
implementations. Added `BucketedAggDataVariants` and associated types for 
bucketed aggregation. 
[[1]](diffhunk://#diff-6ddbe2b1c190d8bc27adcd8942ffc1d7f60f47a463275aa852ea633002436334L50-R63)
 
[[2]](diffhunk://#diff-6ddbe2b1c190d8bc27adcd8942ffc1d7f60f47a463275aa852ea633002436334L69-R203)
   
   **Performance and Memory Improvements:**
   
   * Added reusable buffers for output key storage in hash table contexts to 
reduce per-batch heap allocations and improve performance.
   
   These changes collectively enable efficient, parallel, and memory-aware 
bucketed hash aggregation in the pipeline engine, improving scalability and 
paving the way for further aggregation optimizations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to