[
https://issues.apache.org/jira/browse/IMPALA-5073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16831702#comment-16831702
]
Ruslan Dautkhanov commented on IMPALA-5073:
-------------------------------------------
[~tarmstrong]
My 2 cents. Would it be possible to preallocate a fixed amount of memory for
impalad at startup?
Like it's done in Oracle through `SGA_AGGREGATE_TARGET` (global shared buffers,
cached query plans etc etc) and `PGA_AGGREGATE_TARGET` (for per-session memory
demand like for sorting temp results etc).
This increases startup time, which is okay, but there is no overhead to mmap()
on a per-session / per-query level.
For Oracle SGA always dominates (global db block buffer caches etc), but for
Impala PGA-like memory structure would dominate I guess (per-query shuffling
results etc) .
Another advantage of preallocating memory is that you could use hugetlbfs
optionally as amount of memory for that preallocated memory is static and known
ahead of time.
> Considering bypassing TCMalloc by default for buffer pool
> ---------------------------------------------------------
>
> Key: IMPALA-5073
> URL: https://issues.apache.org/jira/browse/IMPALA-5073
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.9.0
> Reporter: Tim Armstrong
> Priority: Minor
> Labels: resource-management
>
> There would be some advantages to switch from from allocating buffers via
> TCMalloc and instead using mmap directly - e.g. less contention for the page
> heap lock.
> There are also downsides - virtual memory consumption could increase and we
> may end up mapping and unmapping memory more frequently.
> We would also need to wire up the MemTrackers so they include this memory in
> the process estimate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
