[
https://issues.apache.org/jira/browse/IMPALA-12939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Smith reassigned IMPALA-12939:
--------------------------------------
Assignee: Michael Smith
> Improve default for IMPALA_BUILD_THREADS
> ----------------------------------------
>
> Key: IMPALA-12939
> URL: https://issues.apache.org/jira/browse/IMPALA-12939
> Project: IMPALA
> Issue Type: Task
> Components: Infrastructure
> Reporter: Michael Smith
> Assignee: Michael Smith
> Priority: Major
>
> Improve the default selection for IMPALA_BUILD_THREADS and other
> parallelization config.
> Impala's build process needs 2GB of memory per CPU core with {{-notests}} or
> 4GB of memory per CPU core when building unit test binaries (the link process
> is especially memory-intensive). Exceeding these can lead to systems with
> many cores running out of memory during the build. We currently do not
> consider memory when selecting a default value for IMPALA_BUILD_THREADS.
> We currently default IMPALA_BUILD_THREADS to {{{}nproc{}}}, which also does
> not reflect CPU slicing that may happen in containers. How to detect this
> differs between cgroups v1 and v2 (v2 is becoming more common, present on
> Ubuntu 22 and RedHat 9).
> * v1:
> {code:java}
> echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) /
> $(/sys/fs/cgroup/cpu/cpu.cfs_period_us))){code}
> * v2:
> {code:java}
> awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print cores==int(cores)
> ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max{code}
> Something like the following should handle all cases
> {code}
> if [[ -f /sys/fs/cgroup/cpu.max ]]; then
> awk '{ cores = ($1 == "max" ? '$(nproc)' : $1 / $2); print
> cores==int(cores) ? cores : int(cores)+1 }' /sys/fs/cgroup/cpu.max
> elif [[ -f /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]]; then
> echo $(($(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us) /
> $(/sys/fs/cgroup/cpu/cpu.cfs_period_us)))
> else
> nproc
> fi
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
