Re: Support for more than 256 CPU cores
On 5/5/23 6:38 AM, Ed Maste wrote: FreeBSD supports up to 256 CPU cores in the default kernel configuration (on Tier-1 architectures). Systems with more than 256 cores are available now, and will become increasingly common over FreeBSD 14’s lifetime. The FreeBSD Foundation is supporting the effort to increase MAXCPU, and PR269572[1] is open to track tasks and changes. As a project we have scalability work ahead of us to make best use of high core count machines, but at a minimum we should be able to boot a GENERIC kernel on such systems, and have an ABI for the FreeBSD 14 release that supports such a configuration. Some changes have already been committed in support of increased MAXCPU, including increasing MAX_APIC_ID (commit c8113dad7ed4) and a number of changes to reduce bloat (such as commits 42f722e721cd, e72f7ed43eef, 78cfa762ebf2 and 74ac712f72cf). The next step is to increase the maximum cpuset size for userland. I have this change open in review D39941[2] and an exp-run request in PR271213[3]. Following that the kernel change for increasing MAXCPU is in D36838[4]. Additional work on bloat reduction will continue after this change, and looking forward FreeBSD is going to need ongoing effort from the community and the FreeBSD Foundation to continue improving scalability. [1] https://bugs.freebsd.org/269572 [2] https://reviews.freebsd.org/D39941 [3] https://bugs.freebsd.org/271213 [4] https://reviews.freebsd.org/D36838 FWIW, I think it will be useful for main to run with a larger userspace MAXCPU than kernel for at least a while so that we have better testing of that configuration and to give headroom for bumping MAXCPU in the kernel during the 14.x branch. The only other viable path I think which would be more work would be to rework cpuset_t in userspace to always use a dynamically sized mask. This could perhaps be done in an API-preserving manner by making cpuset_t an opaque wrapper type in userland and requiring CPU_* to indirect to functions in libc, etc. That's a fair bit more work however. -- John Baldwin
Re: Support for more than 256 CPU cores
On Sun, May 7, 2023, 3:51 PM Moin Rahman wrote: > > > > On May 5, 2023, at 3:38 PM, Ed Maste wrote: > > > > FreeBSD supports up to 256 CPU cores in the default kernel configuration > > (on Tier-1 architectures). Systems with more than 256 cores are > > available now, and will become increasingly common over FreeBSD 14’s > > lifetime. The FreeBSD Foundation is supporting the effort to increase > > MAXCPU, and PR269572[1] is open to track tasks and changes. > > > > As a project we have scalability work ahead of us to make best use of > > high core count machines, but at a minimum we should be able to boot a > > GENERIC kernel on such systems, and have an ABI for the FreeBSD 14 > > release that supports such a configuration. > > > > Some changes have already been committed in support of increased MAXCPU, > > including increasing MAX_APIC_ID (commit c8113dad7ed4) and a number of > > changes to reduce bloat (such as commits 42f722e721cd, e72f7ed43eef, > > 78cfa762ebf2 and 74ac712f72cf). > > > > The next step is to increase the maximum cpuset size for userland. > > I have this change open in review D39941[2] and an exp-run request in > > PR271213[3]. Following that the kernel change for increasing MAXCPU is > > in D36838[4]. > > > > Additional work on bloat reduction will continue after this change, and > > looking forward FreeBSD is going to need ongoing effort from the > > community and the FreeBSD Foundation to continue improving scalability. > > > > [1] https://bugs.freebsd.org/269572 > > [2] https://reviews.freebsd.org/D39941 > > [3] https://bugs.freebsd.org/271213 > > [4] https://reviews.freebsd.org/D36838 > > > > By any chance will this be merged into 13? > Nope. It breaks too many kernel KBIs that are exposed to at least some modules. Warner > > Kind regards, > Moin >
Re: Support for more than 256 CPU cores
On 5/5/23 17:23, Tomek CEDRO wrote: On Fri, May 5, 2023 at 3:38 PM Ed Maste wrote: FreeBSD supports up to 256 CPU cores in the default kernel configuration (on Tier-1 architectures). Systems with more than 256 cores are available now, and will become increasingly common over FreeBSD 14’s lifetime. (..) Congratulations! :-) I am looking after AMD Threadripper with 64 cores 2 threads each that will give 128 CPU to the system.. maybe this year I could afford that beast then I will report back after testing :-) In upcoming years variations of RISC-V will provide unheard before number of CPU in a single SoC (i.e. 1000 CPU) at amazing power efficiency and I saw reports of prototype with 3 x SoC of this kind on a single board :-) https://spectrum.ieee.org/risc-v-ai Hi, Maybe it makes sense to cluster CPU's in logical groups somehow. Some synchronization mechanism like EPOCH() are O(N²) where N is the number of CPUs. Not in the read-case, but in the synchronize case. It depends a bit though. Currently EPOCH() is executed every kern.hz . --HPS
Re: Support for more than 256 CPU cores
On Fri, May 5, 2023 at 3:38 PM Ed Maste wrote: > FreeBSD supports up to 256 CPU cores in the default kernel configuration > (on Tier-1 architectures). Systems with more than 256 cores are > available now, and will become increasingly common over FreeBSD 14’s > lifetime. (..) Congratulations! :-) I am looking after AMD Threadripper with 64 cores 2 threads each that will give 128 CPU to the system.. maybe this year I could afford that beast then I will report back after testing :-) In upcoming years variations of RISC-V will provide unheard before number of CPU in a single SoC (i.e. 1000 CPU) at amazing power efficiency and I saw reports of prototype with 3 x SoC of this kind on a single board :-) https://spectrum.ieee.org/risc-v-ai -- CeDeROM, SQ7MHZ, http://www.tomek.cedro.info
Support for more than 256 CPU cores
FreeBSD supports up to 256 CPU cores in the default kernel configuration (on Tier-1 architectures). Systems with more than 256 cores are available now, and will become increasingly common over FreeBSD 14’s lifetime. The FreeBSD Foundation is supporting the effort to increase MAXCPU, and PR269572[1] is open to track tasks and changes. As a project we have scalability work ahead of us to make best use of high core count machines, but at a minimum we should be able to boot a GENERIC kernel on such systems, and have an ABI for the FreeBSD 14 release that supports such a configuration. Some changes have already been committed in support of increased MAXCPU, including increasing MAX_APIC_ID (commit c8113dad7ed4) and a number of changes to reduce bloat (such as commits 42f722e721cd, e72f7ed43eef, 78cfa762ebf2 and 74ac712f72cf). The next step is to increase the maximum cpuset size for userland. I have this change open in review D39941[2] and an exp-run request in PR271213[3]. Following that the kernel change for increasing MAXCPU is in D36838[4]. Additional work on bloat reduction will continue after this change, and looking forward FreeBSD is going to need ongoing effort from the community and the FreeBSD Foundation to continue improving scalability. [1] https://bugs.freebsd.org/269572 [2] https://reviews.freebsd.org/D39941 [3] https://bugs.freebsd.org/271213 [4] https://reviews.freebsd.org/D36838
