The parallelism of bitbake easily uses every available core on the build host. But since every task is run with the same number of parallel threads/processes, multiple tasks might load the CPU excessively, which in turn slows down the build due to scheduling overhead.
This patch series adds a class that creates a fifo filled with PARALLEL_MAKE tokens. The path to the created fifo is then configured in the MAKEFLAGS environment variable, which is read by make and a patched ninja (and gcc if doing lto). The benefits from using the jobserver depends on the set of executed tasks: running multiple large do_compile tasks simultaneously benefits more than multiple do_fetch tasks. A simple test building the following tasks (and all their dependencies) yields a ~5% improvement in build time (20:20 -> 19:20): nodejs-native rust-llvm-native rust-native linux-yocto qemu-native On build machines shared by multiple users, a single jobserver can be shared between multiple builds (using the JOBSERVER_FIFO variable). Running the above build in two different build directories at the same time gives a ~12% improvement (43:17 -> 37:55). Finally, the memory pressure from e.g. compiling multiple c++ based projects is also reduced. In our case, a cloud based build machine (with 32 cores and 32GB RAM) fails to compile llvm-rust-native (in parallel to nodejs) without the jobserver due to a lack of memory. This patch set is roughly based on previous work by Richard[1]. That patch lists three TODO items, which are all addressed by these patches: * The fifo path defaults to TMPDIR/jobserver_fifo, but can be configured using JOBSERVER_FIFO. * The number of make threads defaults to the value from PARALLEL_MAKE (which is then redundant). * If PARALLEL_MAKE is unset, the jobserver functionality is skipped. Further work in addition to this patch set could be to make bitbake tasks jobserver aware. Changes since v1[2]: * Fixed typos in various places * Added JOBSERVER_IGNORE variable to jobserver.bbclass in patch 1 * Changed ninja to use kitware fork with only two patches added in patch 3 * Updated upstream patch submission link in qemu in patch 4 Changes since the RFC[3]: * The ninja src uri change in patch 3 is converted to a set of patches * The qemu fix in patch 4 is converted to a submitted patch [1] https://lore.kernel.org/openembedded-core/1423223184.20217.15.ca...@linuxfoundation.org/ [2] https://lore.kernel.org/openembedded-core/20240403070204.367470-1-mar...@geanix.com/ [3] https://lore.kernel.org/openembedded-core/20230828124834.376779-1-mar...@geanix.com/ Martin Hundebøll (5): classes: jobserver: support gnu make fifo jobserver scripts: build-env: allow passing JOBSERVER_FIFO from environment ninja: build modified version with GNU Make jobserver support qemu: enable parallel builds when using the jobserver class contrib: add python service and systemd unit to run shared jobserver contrib/jobserver/jobserver.py | 78 +++++ contrib/jobserver/jobserver.service | 10 + meta/classes-global/jobserver.bbclass | 87 ++++++ meta/conf/bitbake.conf | 2 +- ...ename-TokenPool-Setup-to-SetupClient.patch | 113 ++++++++ ...-jobserver-fifo-style-client-support.patch | 271 ++++++++++++++++++ meta/recipes-devtools/ninja/ninja_1.11.1.bb | 8 +- meta/recipes-devtools/qemu/qemu.inc | 1 + ...e-jobserver-auth-argument-when-calli.patch | 37 +++ scripts/oe-buildenv-internal | 2 +- 10 files changed, 605 insertions(+), 4 deletions(-) create mode 100644 contrib/jobserver/jobserver.py create mode 100644 contrib/jobserver/jobserver.service create mode 100644 meta/classes-global/jobserver.bbclass create mode 100644 meta/recipes-devtools/ninja/files/0001-Rename-TokenPool-Setup-to-SetupClient.patch create mode 100644 meta/recipes-devtools/ninja/files/0002-Add-GNU-make-jobserver-fifo-style-client-support.patch create mode 100644 meta/recipes-devtools/qemu/qemu/0013-Makefile-preserve-jobserver-auth-argument-when-calli.patch -- 2.44.0
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#197955): https://lists.openembedded.org/g/openembedded-core/message/197955 Mute This Topic: https://lists.openembedded.org/mt/105326966/21656 Group Owner: openembedded-core+ow...@lists.openembedded.org Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-