Joe McDonnell created IMPALA-13147: -------------------------------------- Summary: Add support for limiting the concurrency of link jobs Key: IMPALA-13147 URL: https://issues.apache.org/jira/browse/IMPALA-13147 Project: IMPALA Issue Type: Improvement Components: Infrastructure Affects Versions: Impala 4.5.0 Reporter: Joe McDonnell
Link jobs can use a lot of memory due to the amount of debug info. The level of concurrency that is useful for compilation can be too high for linking. Running a link-heavy command like buildall.sh -skiptests can run out of memory from linking all of the backend tests / benchmarks. It would be useful to be able to limit the number of concurrent link jobs. There are two basic approaches: When using the ninja generator for CMake, ninja supports having job pools with limited parallelism. CMake has support for mapping link tasks to their own pool. Here is an example: {noformat} set(CMAKE_JOB_POOLS compilation_pool=24 link_pool=8) set(CMAKE_JOB_POOL_COMPILE compilation_pool) set(CMAKE_JOB_POOL_LINK link_pool){noformat} The makefile generator does not have equivalent functionality, but we could do a more limited version where buildall.sh can split the -skiptests into two make invocations. The first does all the compilation with full parallelism (equivalent to -notests) and then the second make invocation does the backend tests / benchmarks with a reduced parallelism. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org