[
https://issues.apache.org/jira/browse/IMPALA-7446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737902#comment-16737902
]
ASF subversion and git services commented on IMPALA-7446:
---------------------------------------------------------
Commit ae65ff831966eb99417e15738c1aa96013a66f39 in impala's branch
refs/heads/master from Tim Armstrong
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=ae65ff8 ]
IMPALA-7446: enable buffer pool GC when near process mem limit
GC is performed when:
* The amount of memory allocated from the system for the buffer pool
exceeds the reservation (i.e. free buffers and clean pages are not
offset by unused reservation).
* The soft or hard process memory limit would otherwise cause an
allocation to fail.
Testing:
Looped the old version of the semi_joins_exhaustive test, which
reliably reproduced the issue. I confirmed that the buffer pool GC was
running and that it preventing the query failures.
Added a backend test that reproed the issue. A large chunk of the code
change is to add infrastructure to use TCMalloc memory metrics
for the process memory tracker in backend tests.
Ran exhaustive tests.
Change-Id: I81e8e29f1ba319f1b499032f9518d32c511b4b21
Reviewed-on: http://gerrit.cloudera.org:8080/12133
Reviewed-by: Bikramjeet Vig <bikramjeet....@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> Queries can spill earlier than necessary because of accumulation of free
> buffers and clean pages
> ------------------------------------------------------------------------------------------------
>
> Key: IMPALA-7446
> URL: https://issues.apache.org/jira/browse/IMPALA-7446
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 2.10.0, Impala 2.11.0, Impala 3.0, Impala 2.12.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Critical
> Labels: resource-management
> Fix For: Impala 3.2.0
>
>
> See IMPALA-7442 for an example where the query started to spill even when
> memory could have been made available by freeing buffers or evicting clean
> pages. Usually this would just result in spilling earlier than necessary, but
> in the case of IMPALA-7442 it lead to a query failure.
> My original intent was that BufferPool::ReleaseMemory() should be called in
> situations like this, but that was not done.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
