Public bug reported: [Impact]
Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements. Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options. Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel). Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel. Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop- oriented kernel, enabling additional low latency features (LP: #2023007). As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again. Following a detailed analisys of the specific low-latency features: - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time - CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter) - CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-j...@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492) - CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.). Moreover, it is worth noticing that pretty much all the other major Linux distributions are also using CONFIG_HZ0=1000. [Test case] There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive. However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect. We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following. Test case (a CPU-intensive stress test) - stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief Metrics: - measure the bogo ops printed to stdout (not a great metric for real- world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ) Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel. TODO: results will follow. [Fix] Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now). [Regression potential] As already covered we may experience performance regressions in CPU- intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Affects: linux (Ubuntu Noble) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Noble) Importance: Undecided Status: New ** Description changed: [Impact] Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements. Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options. Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel). Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel. Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop- oriented kernel, enabling additional low latency features (LP: #2023007). As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again. Following a detailed analisys of the specific low-latency features: - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time - - CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to + - CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter) - - CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a + - CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-j...@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492) - - CONFIG_HZ=1000 last but not least, the only option that is *only* + - CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU-bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications - (gaming, live conferencing, multimedia, etc.). + (gaming, live conferencing, multimedia, etc.). Moreover, it is worth + noticing that pretty much all the other major Linux distributions are + also using CONFIG_HZ0=1000. [Test case] There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive. However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect. We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following. Test case (a CPU-intensive stress test) - - stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m + - stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief Metrics: - - measure the bogo ops printed to stdout (not a great metric for real- + - measure the bogo ops printed to stdout (not a great metric for real- world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ) Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel. TODO: results will follow. [Fix] Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now). [Regression potential] As already covered we may experience performance regressions in CPU- intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/2051342 Title: Enable lowlatency settings in the generic kernel Status in linux package in Ubuntu: New Status in linux source package in Noble: New Bug description: [Impact] Ubuntu provides the "lowlatency" kernel: a kernel optimized for applications that have special "low latency" requirements. Currently, this kernel does not include any specific UBUNTU SAUCE patches to improve the extra "low latency" requirements, but the only difference is a small subset of .config options. Almost all these options are now configurable either at boot-time or even at run-time, with the only exception of CONFIG_HZ (250 in the generic kernel vs 1000 in the lowlatency kernel). Maintaining a separate kernel for a single config option seems a bit overkill and it is a significant cost of engineering hours, build time, regression testing time and resources. Not to mention the risk of the low-latency kernel falling behind and not being perfectly in sync with the latest generic kernel. Enabling the low-latency settings in the generic kernel has been evaluated before, but it has been never finalized due to the potential risk of performance regressions in CPU-intensive applications (increasing HZ from 250 to 1000 may introduce more kernel jitter in number crunching workloads). The outcome of the original proposal resulted in a re-classification of the lowlatency kernel as a desktop- oriented kernel, enabling additional low latency features (LP: #2023007). As we are approaching the release of the new Ubuntu 24.04 we may want to re-consider merging the low-latency settings in the generic kernel again. Following a detailed analisys of the specific low-latency features: - CONFIG_NO_HZ_FULL=y: enable access to "Full tickless mode" (shutdown clock tick when possible across all the enabled CPUs if they are either idle or running 1 task - reduce kernel jitter of running tasks due to the periodic clock tick, must be enabled at boot time passing `nohz_full=<cpu_list>`); this can actually help CPU-intensive workloads and it could provide much more benefits than the CONFIG_HZ difference (since it can potentially shutdown any kernel jitter on specific CPUs), this one should really be enabled anyway, considering that it is configurable at boot time - CONFIG_RCU_NOCB_CPU=y: move RCU callbacks from softirq context to kthread context (reduce time spent in softirqs with preemption disabled to improve the overall system responsiveness, at the cost of introducing a potential performance penalty, because RCU callbacks are not processed by kernel threads); this should be enabled as well, since it is configurable at boot time (via the rcu_nocbs=<cpu_list> parameter) - CONFIG_RCU_LAZY=y: batch RCU callbacks and then flush them after a timed delay instead of executing them immediately (can provide 5~10% power-savings for idle or lightly-loaded systems, this is extremely useful for laptops / portable devices - https://lore.kernel.org/lkml/20221016162305.2489629-3-j...@joelfernandes.org/); this has the potential to introduce significant performance regressions, but in the Noble kernel we already have a SAUCE patch that allows to enable/disable this option at boot time (see LP: #2045492) - CONFIG_HZ=1000 last but not least, the only option that is *only* tunable at compile time. As already mentioned there is a potential risk of regressions for CPU-intensive applications, but they can be mitigated (and maybe they could even outperformed) with NO_HZ_FULL. On the other hand, HZ=1000 can improve system responsiveness, that means most of the desktop and server applications will benefit from this (the largest part of the server workloads is I/O bound, more than CPU- bound, so they can benefit from having a kernel that can react faster at switching tasks), not to mention the benefit for the typical end users applications (gaming, live conferencing, multimedia, etc.). Moreover, it is worth noticing that pretty much all the other major Linux distributions are also using CONFIG_HZ0=1000. [Test case] There are plenty of micro-benchmarks to prove the validity of each one of the config mentioned above, in terms of system responsive. However, our main goal here is to mitigate as much as possible the risk of regression for CPU-intensive applications, so the test case should be focused on this particular aspect. We also have the opportunity to test these changes using the lowlatency kernel, therefore a reasonable test plan could be defined as following. Test case (a CPU-intensive stress test) - stress-ng --matrix $(getconf _NPROCESSORS_ONLN) --timeout 5m --metrics-brief Metrics: - measure the bogo ops printed to stdout (not a great metric for real-world applications, but in this case they can show the impact of the addditional kernel jitter introduced by the different CONFIG_HZ) Perform multiple runs of the stress test, both on amd64 and arm64. Repeat the test also with CONFIG_NO_HZ_FULL. If NO_HZ_FULL can provide better results than the CONFIG_HZ=250 case, we can safely switch to CONFIG_HZ=1000, since we have the possibility to provide a boot-time option to improve CPU-intensive workloads and get better performance than the previous kernel. TODO: results will follow. [Fix] Enable the .config options mentioned above in the generic kernel (only on amd64 and arm64 for now). [Regression potential] As already covered we may experience performance regressions in CPU- intensive (number crunching) applications (such as HPC for example), but they can be compensated by the NO_HZ_FULL boot-time option. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2051342/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
