[
https://issues.apache.org/jira/browse/IMPALA-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17797276#comment-17797276
]
ASF subversion and git services commented on IMPALA-11805:
----------------------------------------------------------
Commit 6cc8114813eea18810c4fc4109ef06e63a23e077 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6cc811481 ]
Revert "IMPALA-11805: Fix LLVM memory manager bytes allocated"
This reverts commit 8874ea07b68c09cee3c4e300b70706f3c5ea40af.
Reason for revert: other work on IMPALA-11805 will stop using the memory
manager's size for the codegen cache, and changing this had broader
implications on query memory tracking that need to be accounted for. It
started causing memory reservation failures in tests.
Change-Id: I7f7a6b21b3a9fc7c9e675c8d3349725c4863f744
Reviewed-on: http://gerrit.cloudera.org:8080/20796
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Michael Smith <michael.sm...@cloudera.com>
> Codegen cache size estimation is less than the actual allocation
> ----------------------------------------------------------------
>
> Key: IMPALA-11805
> URL: https://issues.apache.org/jira/browse/IMPALA-11805
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.3.0
> Reporter: Yida Wu
> Assignee: Yida Wu
> Priority: Major
>
> In IMPALA-11470, we implement the cache for codegen functions, however, the
> expected size of a cache entry is much less than the actual allocation,
> according to the data in tcmalloc memory tracker. This could lead to a result
> of unexpected query failure when the memory tracker hits the capacity.
> The current way to estimate the memory consumption of a codegen cache entry,
> mainly the memory consumption of a llvm::ExecutionEngine that stored in each
> entry, is to use the customized ImpalaMCJITMemoryManager
> [https://github.com/apache/impala/blob/f705496e34ac474e8e1c999619e3b928c5e39e0f/be/src/codegen/mcjit-mem-mgr.h#L60],
> to accumulated bytes when the execution engine allocates code or data
> section. However in fact, the actual bytes allocated by the execution engine
> could be much larger.
> Tested in tpch and tpcds queries, in normal mode, the final consumption could
> be 3~4 times of the estimation, and it would be worse in the optimal mode,
> because the main difference is between the memory_manager_->bytes_allocated()
> and the actual execution engine allocation, and in normal mode it contains
> the size of the key, which is accurate.
> When the execution engine is only existing a short period in runtime, the
> issue isn't that bad. However, when it becomes a part of the long-living
> cache entry, it could cause more problems by consuming much more memory than
> estimation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
