[
https://issues.apache.org/jira/browse/DRILL-6032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16497241#comment-16497241
]
ASF GitHub Bot commented on DRILL-6032:
---------------------------------------
ilooner commented on issue #1101: DRILL-6032: Made the batch sizing for HashAgg
more accurate.
URL: https://github.com/apache/drill/pull/1101#issuecomment-393694572
While making this change I learned that it is extremely difficult to make
changes to the HashAgg operator, specifically the memory calculations. After a
month of work I was not able to bring even a simple change like this to a state
where it would not OOM. Others experience similar frustrations when they make
changes that either impact the HashAgg operator directly, or indirectly.
Considering the fact that a comprehensive and stable memory calculator was
created for HashJoin in less than a month, I feel that there is an easier way
forward.
As a result of this experience I was motivated to create
https://issues.apache.org/jira/browse/DRILL-6253 which focuses on refactoring
and unit testing the memory calculations for HashAgg . DRILL-6253 will fulfill
the original intention of this PR, which was to make the memory calculations
more accurate, and also improve the general stability and testability of
HashAgg. As a result of this, I am going to abandon this PR and all future work
will be done on the PR for DRILL-6253 once it is opened.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Use RecordBatchSizer to estimate size of columns in HashAgg
> -----------------------------------------------------------
>
> Key: DRILL-6032
> URL: https://issues.apache.org/jira/browse/DRILL-6032
> Project: Apache Drill
> Issue Type: Improvement
> Reporter: Timothy Farkas
> Assignee: Timothy Farkas
> Priority: Major
> Fix For: 1.15.0
>
>
> We need to use the RecordBatchSize to estimate the size of columns in the
> Partition batches created by HashAgg.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
