[
https://issues.apache.org/jira/browse/IMPALA-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17062324#comment-17062324
]
ASF subversion and git services commented on IMPALA-9530:
---------------------------------------------------------
Commit ca53f68525504b6a1a64bb35c7edb57d030916d7 in impala's branch
refs/heads/master from Tim Armstrong
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=ca53f68 ]
IMPALA-9530: query option to limit preagg memory
This adds an advanced PREAGG_BYTES_LIMIT query option that
allows limiting the memory consumption of streaming
preaggregation operators in a query.
It works by setting a maximum reservation on each grouping
aggregator in a preaggregation node. The aggregators switch
to passthrough mode automatically when hitting this limit,
the same as if they were hitting the query memory limit.
This does not override the minimum reservation computed for
the aggregation - if the limit is less than the minimum
reservation, the minimum reservation is used as the limit
instead.
The default behaviour is unchanged.
Testing:
Add a planner test with estimates higher and lower than limit
to ensure that resource estimates correctly reflect the option.
Add an end-to-end test that verifies that the option forces
passthrough when the memory limit is hit.
Change-Id: I87f7a5c68da93d068e304ef01afbcbb0d56807d9
Reviewed-on: http://gerrit.cloudera.org:8080/15463
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Allow limiting memory consumed by preaggregation
> ------------------------------------------------
>
> Key: IMPALA-9530
> URL: https://issues.apache.org/jira/browse/IMPALA-9530
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
>
> In some cases pre-aggregations can balloon up and consume lots of memory,
> forcing the merge aggregation to spill. This is often OK as long as the
> preaggregation is reducing the input sufficiently, since it reduces the
> amount of data shuffled over the network.
> However in some cases it's preferable to be more conservative with memory and
> just cap the size of the preaggregation to prevent it ballooning too much.
> It would be useful to add a query option to directly limit the memory
> consumption of the preaggregations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
