Re: Support for more than 256 CPU cores

2023-05-05 Thread Hans Petter Selasky 

On 5/5/23 17:23, Tomek CEDRO wrote:

On Fri, May 5, 2023 at 3:38 PM Ed Maste wrote:

FreeBSD supports up to 256 CPU cores in the default kernel configuration
(on Tier-1 architectures).  Systems with more than 256 cores are
available now, and will become increasingly common over FreeBSD 14’s
lifetime. (..)


Congratulations! :-)

I am looking after AMD Threadripper with 64 cores 2 threads each that
will give 128 CPU to the system.. maybe this year I could afford that
beast then I will report back after testing :-)

In upcoming years variations of RISC-V will provide unheard before
number of CPU in a single SoC (i.e. 1000 CPU) at amazing power
efficiency and I saw reports of prototype with 3 x SoC of this kind on
a single board :-)

https://spectrum.ieee.org/risc-v-ai



Hi,

Maybe it makes sense to cluster CPU's in logical groups somehow. Some 
synchronization mechanism like EPOCH() are O(N²) where N is the number 
of CPUs. Not in the read-case, but in the synchronize case. It depends a 
bit though. Currently EPOCH() is executed every kern.hz .


--HPS

Re: Support for more than 256 CPU cores

2023-05-05 Thread Tomek CEDRO 
On Fri, May 5, 2023 at 3:38 PM Ed Maste wrote:
> FreeBSD supports up to 256 CPU cores in the default kernel configuration
> (on Tier-1 architectures).  Systems with more than 256 cores are
> available now, and will become increasingly common over FreeBSD 14’s
> lifetime. (..)

Congratulations! :-)

I am looking after AMD Threadripper with 64 cores 2 threads each that
will give 128 CPU to the system.. maybe this year I could afford that
beast then I will report back after testing :-)

In upcoming years variations of RISC-V will provide unheard before
number of CPU in a single SoC (i.e. 1000 CPU) at amazing power
efficiency and I saw reports of prototype with 3 x SoC of this kind on
a single board :-)

https://spectrum.ieee.org/risc-v-ai

-- 
CeDeROM, SQ7MHZ, http://www.tomek.cedro.info

Support for more than 256 CPU cores

2023-05-05 Thread Ed Maste 
FreeBSD supports up to 256 CPU cores in the default kernel configuration
(on Tier-1 architectures).  Systems with more than 256 cores are
available now, and will become increasingly common over FreeBSD 14’s
lifetime.  The FreeBSD Foundation is supporting the effort to increase
MAXCPU, and PR269572[1] is open to track tasks and changes.

As a project we have scalability work ahead of us to make best use of
high core count machines, but at a minimum we should be able to boot a
GENERIC kernel on such systems, and have an ABI for the FreeBSD 14
release that supports such a configuration.

Some changes have already been committed in support of increased MAXCPU,
including increasing MAX_APIC_ID (commit c8113dad7ed4) and a number of
changes to reduce bloat (such as commits 42f722e721cd, e72f7ed43eef,
78cfa762ebf2 and 74ac712f72cf).

The next step is to increase the maximum cpuset size for userland.
I have this change open in review D39941[2] and an exp-run request in
PR271213[3].  Following that the kernel change for increasing MAXCPU is
in D36838[4].

Additional work on bloat reduction will continue after this change, and
looking forward FreeBSD is going to need ongoing effort from the
community and the FreeBSD Foundation to continue improving scalability.

[1] https://bugs.freebsd.org/269572
[2] https://reviews.freebsd.org/D39941
[3] https://bugs.freebsd.org/271213
[4] https://reviews.freebsd.org/D36838

Re: 20230504 -CURRENT snapshot panics during install at zfs probing

2023-05-05 Thread Rebecca Cran 
I also tried FreeBSD 13.2 and it's unhappy too, and ends up panicing and 
rebooting when trying to restart the installer after it segfaults. So 
it's apparently _not_ a new problem.


I'm guessing there's something about my disks that's causing memory 
corruption. The only thing that's changed recently is installing Windows 
Server 2022 on a different disk.


If I disable the mrsas controller, installation proceeds.


--
Rebecca Cran


On 5/4/23 19:02, Rebecca Cran wrote:
I'm seeing a panic during install during the zfs probing on the 
14-CURRENT 20230504 amd64 snapshot on a PowerEdge R7525 dual-EPYC system.


I haven't submitted a bug report for a while, so let me know what 
other information is needed.



Unfortunately the stack backtrace looks useless:

panic: mtx_lock() of spin mutex (null) @ 
/usr/src/sys/kern/kern_clock.c:269

cpuid = 67
time = 1683225642
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 
0xfe037aa72db0

vpanic() at vpanic+0x152/frame 0xfe037aa72e00
panic() at panic+0x43/frame 0xfe037aa72e60
__mtx_lock_flags() at __mtx_lock_flags+0x13b/frame 0xfe037aa72eb0
deadlkres() at deadlkres+0xef/frame 0xfe037aa72ef0
fork_exit() at fork_exit+0x80/frame 0xfe037aa72f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfe037aa72f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 0 tid 101238 ]
Stopped at  kdb_enter+0x32: movq    $0,0xddd833(%rip)
db>


Start of dmesg:

FreeBSD 14.0-CURRENT #0 main-n262746-4194bbb34c60: Thu May  4 08:05:46 
UTC 2023
r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC 
amd64
FreeBSD clang version 15.0.7 (https://github.com/llvm/llvm-project.git 
llvmorg-15.0.7-0-g8dfdcc7b7bf6)

WARNING: WITNESS option enabled, expect reduced performance.
SRAT: Too many memory domains
VT(efifb): resolution 1024x768
CPU: AMD EPYC 7713 64-Core Processor (1996.32-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0xa00f11  Family=0x19  Model=0x1 Stepping=1
Features=0x178bfbff 

Features2=0x7efa320b 


  AMD Features=0x2e500800
  AMD 
Features2=0x75c237ff
  Structured Extended 
Features=0x219c97a9
  Structured Extended 
Features2=0x40069c

  Structured Extended Features3=0x10
  XSAVE Features=0xf
  AMD Extended Feature Extensions ID 
EBX=0x91bef75f

  SVM: NP,NRIP,VClean,AFlush,DAssist,NAsids=32768
  TSC: P-state invariant, performance statistics
real memory  = 277021196288 (264188 MB)
avail memory = 266164191232 (253833 MB)

disk info:

root@:~ # gpart show
=>    6  233308149  nvd0  GPT  (890G)
  6  233308149    - free -  (890G)

=>    6  233308149  diskid/DISK-S61ANA0R100142  GPT  (890G)
  6  233308149  - free - (890G)

=>   34  937571901  ada0  GPT  (447G)
 34   2014    - free -  (1.0M)
   2048  937568256 1  ms-basic-data  (447G)
  937570304   1631    - free -  (816K)

=>   34  937571901  diskid/DISK-74ff0e529b990010  GPT (447G)
 34   2014    - free - (1.0M)
   2048  937568256 1 ms-basic-data  
(447G)

  937570304   1631    - free - (816K)

=>    34  3125627501  da0  GPT  (1.5T)
  34    2014   - free -  (1.0M)
    2048 1048576    1  efi  (512M)
 1050624  2929686528    2  linux-data  (1.4T)
  2930737152    62500864    3  linux-swap  (30G)
  2993238016   132389519   - free -  (63G)

=>    34  3125627501  da1  GPT  (1.5T)
  34    2014   - free -  (1.0M)
    2048   32768    1  ms-reserved  (16M)
   34816  3125592064    2  ms-basic-data  (1.5T)
  3125626880 655   - free -  (328K)

=>    34  3125627501  diskid/DISK-S61ENE0R302148  GPT (1.5T)
  34    2014  - free - (1.0M)
    2048 1048576   1  efi  (512M)
 1050624  2929686528   2  linux-data (1.4T)
  2930737152    62500864   3  linux-swap (30G)
  2993238016   132389519  - free - (63G)

=>    34  3125627501  diskid/DISK-202229EBAC90  GPT  (1.5T)
  34    2014    - free - (1.0M)
    2048   32768 1  ms-reserved (16M)
   34816  3125592064 2  ms-basic-data (1.5T)
  3125626880 655    - free - (328K)

=>    17  544157  cd0  MBR  (1.0G)
  17  544157   - free -  (1.0G)

=>    17  544157  iso9660/14_0_CURRENT_AMD64_CD  MBR  (1.0G)
  17  544157 - free -  (1.0G)