[
https://issues.apache.org/jira/browse/IMPALA-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609864#comment-16609864
]
Tim Armstrong edited comment on IMPALA-7446 at 12/27/18 6:09 PM:
-----------------------------------------------------------------
I think the precondition for GC of free buffers and clean pages should be both:
* Buffer allocations exceed reservation
* We're at or near the process memory limit
We should also be counting reserved-but-not-allocated memory against the
process memory limit when considering whether to let other memory allocations
go through. Otherwise a future allocation based on a reservation (which doesn't
check the process limit, by design) could go through even though it would push
the impalad above the process limit.
edit: this second point is now IMPALA-8020
was (Author: tarmstrong):
I think the precondition for GC of free buffers and clean pages should be both:
* Buffer allocations exceed reservation
* We're at or near the process memory limit
We should also be counting reserved-but-not-allocated memory against the
process memory limit when considering whether to let other memory allocations
go through. Otherwise a future allocation based on a reservation (which doesn't
check the process limit, by design) could go through even though it would push
the impalad above the process limit.
> Queries can spill earlier than necessary because of accumulation of free
> buffers and clean pages
> ------------------------------------------------------------------------------------------------
>
> Key: IMPALA-7446
> URL: https://issues.apache.org/jira/browse/IMPALA-7446
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: resource-management
>
> See IMPALA-7442 for an example where the query started to spill even when
> memory could have been made available by freeing buffers or evicting clean
> pages. Usually this would just result in spilling earlier than necessary, but
> in the case of IMPALA-7442 it lead to a query failure.
> My original intent was that BufferPool::ReleaseMemory() should be called in
> situations like this, but that was not done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
