Re: Add support for limiting CPU pressure
On Wed, 2022-12-21 at 15:00 -0500, Randy MacLeod wrote: > What tests were you running to check these improvements? The GNU project doesn't have the resources to perform large-scale testing. We run our regression tests before pushing any code, but that's about all we can manage. The only regular testing we do of -l is to ensure it works (we set the load so low that we're guaranteed to get serialized builds regardless of -j, and make sure that's so). Before pulling these changes in I did some straightforward testing myself, by simply running a large build (GCC IIRC) and verifying, by spot-checking, that we never had too many jobs running. I'm confident I didn't try running multiple disjoint invocations of make in parallel to see what the load was like. I'm always excited to get more testing! Either from the Git repository, or especially during the release candidate cycle: all RC builds are announced on this list and anyone who has the facilities to stress-test the RC builds would be most appreciated!
Re: Add support for limiting CPU pressure
On 2022-12-21 11:24, Paul Smith wrote: On Tue, 2022-12-20 at 16:59 -0500, Randy MacLeod wrote: While we are doing development with Yocto Project (1), a source- based Linux distro builder, we found that many instances of making running in parallel with other work can overload even a large many-core build machine. Existing mechanisms that are either not system-wide (-j) or are too slow(-l has 1 minute averaging), so in order to make optimal use of a multi-core system for some larger tasks we need a new mechanism. Can you clarify what version of GNU make you're using? 4.3 from ubuntu-22.04 but the build pool for Yocto has ~15 distros of various vintages as well as being able to use the version of make (4.4) that we carry on the oe-core master branch: https://layers.openembedded.org/layerindex/recipe/245/ http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-devtools/make/make_4.4.bb For any work we do here, we'd use that master branch version rather than the older host version. There have been a number of efforts to improve the accuracy of -l and it does not (any longer) use a simple load detection, for EXACTLY this reason (the value is too coarse, particularly at startup). Okay. I'll try to allocate time to see what happens with 4.4. In the latest version GNU make 4.4, it will use the content of /proc/loadavg (if it exists) to figure out how many jobs are currently executing on the system and use that as a measurement of load. If that's not available, we use getloadavg() but we do some math on it to modify the load average based on the number of jobs we've started in the last second. Of course, this also suffers from the problem that it only knows about jobs that THIS INSTANCE of make has started in the last second. Anyway you can read more about this in the comments in the source code: https://git.savannah.gnu.org/cgit/make.git/tree/src/job.c?h=4.4#n1953 Thanks, I missed the recent changes amongst the decades old code! ;-) What tests were you running to check these improvements? -- # Randy MacLeod # Wind River Linux
Re: Add support for limiting CPU pressure
On Tue, 2022-12-20 at 16:59 -0500, Randy MacLeod wrote: > > While we are doing development with Yocto Project (1), a source- > > based Linux distro builder, we found that many instances of making > > running in parallel with other work can overload even a large > > many-core build machine. Existing mechanisms that are either not > > system-wide (-j) or are too slow(-l has 1 minute averaging), so in > > order to make optimal use of a multi-core system for some larger > > tasks we need a new mechanism. Can you clarify what version of GNU make you're using? There have been a number of efforts to improve the accuracy of -l and it does not (any longer) use a simple load detection, for EXACTLY this reason (the value is too coarse, particularly at startup). In the latest version GNU make 4.4, it will use the content of /proc/loadavg (if it exists) to figure out how many jobs are currently executing on the system and use that as a measurement of load. If that's not available, we use getloadavg() but we do some math on it to modify the load average based on the number of jobs we've started in the last second. Of course, this also suffers from the problem that it only knows about jobs that THIS INSTANCE of make has started in the last second. Anyway you can read more about this in the comments in the source code: https://git.savannah.gnu.org/cgit/make.git/tree/src/job.c?h=4.4#n1953
Re: Add support for limiting CPU pressure
Re-send now that I've subscribed from my work email as well as my gmail account. Thanks for the comment Howard. On 2022-12-20 16:02, Howard Chu wrote: cont...@zhengqiu.net wrote: Hello, While we are doing development with Yocto Project (1), a source- based Linux distro builder, we found that many instances of making running in parallel with other work can overload even a large many-core build machine. Existing mechanisms that are either not system-wide (-j) or are too slow(-l has 1 minute averaging), so in order to make optimal use of a multi-core system for some larger tasks we need a new mechanism. All the tests are done using an Ubuntu System, as we are not very lucky to find similar features on macOS/Windows, so if you know anything, please help us out! Additionally, we also want to gather more data, so if you know any other large packages that make is often tested with, please let us know as well! Relying on a new non-portable kernel feature sounds like a pretty bad idea. Especially when you can easily solve the "not system-wide" aspect of "make -j" portably: by using a named pipe. It should be a simple job to patch make to create or reference a named pipe in an initial invocation, and the rest of the existing jobserver machinery will Just Work after that, and all of the relevant OSs have support for named pipes. We did consider this approach and it can work in some but certainly not all use cases and not the ones where this approach has the most impact. It could work where we "only" have one bitbake build (bitbake is the build tool used in the yocto world) running and we want to share jobs across this single bitbake build. We'd still have a problem with recipes (packages) that do not use make to build and that don't have a job server design. While we could add that feature to the other build tools such as ninja, it seems that PSI is a more generic solution. What this approach doesn't deal with occurs when there are several bitbake builds that are all independently running or a situation where there are say a couple of bitbake builds, a CPU intensive runtime test using qemu and a clean-up job all running at the same time. That may seem like an unusual use-case but the generic version is that you want to do a build but you want to back-off if other processes in any account, are also loading the machine that you are using. If several users or builds are all using the same PSI back-off mechanism then the system will be less likely to get into an overload or even a process thrashing situation. The shared job server approach is a good idea but has more limited applicability, right? Also, I'm re-posting the original thread below for the archives since Zheng's email wasn't plain text and didn't get archived properly. Oh and I have one comment below for Zheng. I'll trim the thread on any follow-up. Thanks, ../Randy Hello, While we are doing development with Yocto Project (1), a source- based Linux distro builder, we found that many instances of making running in parallel with other work can overload even a large many-core build machine. Existing mechanisms that are either not system-wide (-j) or are too slow(-l has 1 minute averaging), so in order to make optimal use of a multi-core system for some larger tasks we need a new mechanism. We found that on Linux, for the 4.20 kernel and later, a feature called Pressure Stall Information (PSI(2)) can provide system-wide metrics indicating when and by how much a system is experiencing cpu, memory or io pressure. So we implemented a new feature that uses /proc/pressure/cpu info to limit new task creation. We previously implemented it for bitbake: , and find limiting tasks by using proc/pressure/cpu can significantly reduce system latency and CPU contention after Yocto uses this feature from bitbake, their CPU contention-related errors have been reduced to about once every two months compared to several times every week. Here is the commit we have, although it needs to be cleaned up before commit, it works fine when we tested it with OpenSSL on a 4-core system and found the CPU pressure can be reduced by 20% while keeping the run time about the same. This is not ideal though, so we also want to see if you have any suggested improvements to this algorithm. $ hyperfine --runs 5 'make clean && /usr/bin/time -o build-time-0.log ./../mymake -j' Benchmark 1: make clean && /usr/bin/time -o build-time-0.log ./../mymake -j Time (mean ± σ): 179.994 s ± 0.418 s[User: 576.071 s, System: 56.747 s] Range (min … max): 179.383 s … 180.441 s5 runs $ hyperfine --runs 5 'make clean && /usr/bin/time -o build-time-0.log ./../mymake -j -z 10' Benchmark 1: make clean && /usr/bin/time -o build-time-0.log ./../mymake -j -z 10 Time (mean ± σ): 166.372 s ± 4.976 s[User: 538.634 s, System: 59.617 s] Range (min … max): 159.443 s … 171.906 s5 runs $ hyperfine --runs 5 'make clean && /usr/bin/time -o
Re: Add support for limiting CPU pressure
Thanks for the comment Howard. On 2022-12-20 16:02, Howard Chu wrote: cont...@zhengqiu.net wrote: Hello, While we are doing development with Yocto Project (1), a source- based Linux distro builder, we found that many instances of making running in parallel with other work can overload even a large many-core build machine. Existing mechanisms that are either not system-wide (-j) or are too slow(-l has 1 minute averaging), so in order to make optimal use of a multi-core system for some larger tasks we need a new mechanism. All the tests are done using an Ubuntu System, as we are not very lucky to find similar features on macOS/Windows, so if you know anything, please help us out! Additionally, we also want to gather more data, so if you know any other large packages that make is often tested with, please let us know as well! Relying on a new non-portable kernel feature sounds like a pretty bad idea. Especially when you can easily solve the "not system-wide" aspect of "make -j" portably: by using a named pipe. It should be a simple job to patch make to create or reference a named pipe in an initial invocation, and the rest of the existing jobserver machinery will Just Work after that, and all of the relevant OSs have support for named pipes. We did consider this approach and it can work in some but certainly not all use cases and not the ones where this approach has the most impact. It could work where we "only" have one bitbake build (bitbake is the build tool used in the yocto world) running and we want to share jobs across this single bitbake build. We'd still have a problem with recipes (packages) that do not use make to build and that don't have a job server design. While we could add that feature to the other build tools such as ninja, it seems that PSI is a more generic solution. What this approach doesn't deal with occurs when there are several bitbake builds that are all independently running or a situation where there are say a couple of bitbake builds, a CPU intensive runtime test using qemu and a clean-up job all running at the same time. That may seem like an unusual use-case but the generic version is that you want to do a build but you want to back-off if other processes in any account, are also loading the machine that you are using. If several users or builds are all using the same PSI back-off mechanism then the system will be less likely to get into an overload or even a process thrashing situation. The shared job server approach is a good idea but has more limited applicability, right? Also, I'm re-posting the original thread below for the archives since Zheng's email wasn't plain text and didn't get archived properly. Oh and I have one comment below for Zheng. I'll trim the thread on any follow-up. Thanks, ../Randy Hello, While we are doing development with Yocto Project (1), a source- based Linux distro builder, we found that many instances of making running in parallel with other work can overload even a large many-core build machine. Existing mechanisms that are either not system-wide (-j) or are too slow(-l has 1 minute averaging), so in order to make optimal use of a multi-core system for some larger tasks we need a new mechanism. We found that on Linux, for the 4.20 kernel and later, a feature called Pressure Stall Information (PSI(2)) can provide system-wide metrics indicating when and by how much a system is experiencing cpu, memory or io pressure. So we implemented a new feature that uses /proc/pressure/cpu info to limit new task creation. We previously implemented it for bitbake: , and find limiting tasks by using proc/pressure/cpu can significantly reduce system latency and CPU contention after Yocto uses this feature from bitbake, their CPU contention-related errors have been reduced to about once every two months compared to several times every week. Here is the commit we have, although it needs to be cleaned up before commit, it works fine when we tested it with OpenSSL on a 4-core system and found the CPU pressure can be reduced by 20% while keeping the run time about the same. This is not ideal though, so we also want to see if you have any suggested improvements to this algorithm. $ hyperfine --runs 5 'make clean && /usr/bin/time -o build-time-0.log ./../mymake -j' Benchmark 1: make clean && /usr/bin/time -o build-time-0.log ./../mymake -j Time (mean ± σ): 179.994 s ± 0.418 s[User: 576.071 s, System: 56.747 s] Range (min … max): 179.383 s … 180.441 s5 runs $ hyperfine --runs 5 'make clean && /usr/bin/time -o build-time-0.log ./../mymake -j -z 10' Benchmark 1: make clean && /usr/bin/time -o build-time-0.log ./../mymake -j -z 10 Time (mean ± σ): 166.372 s ± 4.976 s[User: 538.634 s, System: 59.617 s] Range (min … max): 159.443 s … 171.906 s5 runs $ hyperfine --runs 5 'make clean && /usr/bin/time -o build-time-0.log ./../mymake -j -z 50' Benchmark 1: make clean && /usr/bin/time -o
Re: Add support for limiting CPU pressure
cont...@zhengqiu.net wrote: > Hello, > > While we are doing development with Yocto Project (1), a source- > based Linux distro builder, we found that many instances of making > running in parallel with other work can overload even a large > many-core build machine. Existing mechanisms that are either not > system-wide (-j) or are too slow(-l has 1 minute averaging), so in > order to make optimal use of a multi-core system for some larger > tasks we need a new mechanism. > All the tests are done using an Ubuntu System, as we are not very lucky to > find similar features on macOS/Windows, so if you know anything, please > help us out! Additionally, we also want to gather more data, so if you know > any other large packages that make is often tested with, please let us know > as well! Relying on a new non-portable kernel feature sounds like a pretty bad idea. Especially when you can easily solve the "not system-wide" aspect of "make -j" portably: by using a named pipe. It should be a simple job to patch make to create or reference a named pipe in an initial invocation, and the rest of the existing jobserver machinery will Just Work after that, and all of the relevant OSs have support for named pipes. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
