Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24961
Change subject: IMPALA-13329: Add cgroup cpuset support to single_node_perf_run.py ...................................................................... IMPALA-13329: Add cgroup cpuset support to single_node_perf_run.py The perf-AB-test jobs currently run multiple Impalads on a single machine with no isolation. The Impalads and minicluster can run on all CPUs. This allows cross-talk between different Impalads and the minicluster, which makes measurements noisier. To reduce the noise level, this introduces a mode (--use_cgroup_cpusets) that uses cgroup cpusets to isolate Impalads to their own CPUs. This creates a cgroup per Impalad and an "other" cgroup for non-Impalad processes (e.g. the runner script and minicluster). Each Impalad is given --cpus_per_impalad CPUs. All remaining CPUs are given to the "other" cgroup. There is also an option to disable hyperthreading by mapping Impalads to only one out of the two hyperthreads. This only supports Cgroups V2. Cgroups V2 has stricter rules about processes creating and managing their own cgroups. To avoid running afoul of those rules, this runs single_node_perf_run.py via a systemd service with cgroup delegation enabled. This means that single_node_perf_run.py starts in a base cgroup and has permissions to create subgroups for the Impalads and minicluster. Systemd will clean this hierarchy up when the service exits. Since this is somewhat involved, this adds a bin/jenkins/perf-ab-test.sh runner script to use for Jenkins jobs. Testing: - Ran the regular non-cgroup perf-AB-test job - Ran a new job that uses cgroup cpusets on x86_64 and ARM Change-Id: I80e5a6cc9a2afe9b603c84eb84e728cf14ea6944 --- A bin/jenkins/perf-ab-test.sh M bin/single_node_perf_run.py 2 files changed, 527 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/61/24961/1 -- To view, visit http://gerrit.cloudera.org:8080/24961 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I80e5a6cc9a2afe9b603c84eb84e728cf14ea6944 Gerrit-Change-Number: 24961 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell <[email protected]>
