On Thu, Jun 8, 2023 at 6:59 PM Yu Zhao wrote:
>
> TLDR
>
> Multichase in 64 microVMs achieved 6% more total samples (in ~4 hours) after
> this patchset [1].
>
> Hardware
>
> HOST $ lscpu
> Architecture:x86_64
> CPU op-mode(s):32-bit, 64-bit
> Address sizes: 43 bits physical, 48 bits virtual
> Byte Order:Little Endian
> CPU(s): 128
> On-line CPU(s) list: 0-127
> Vendor ID: AuthenticAMD
> Model name:AMD Ryzen Threadripper PRO 3995WX 64-Cores
> CPU family: 23
> Model: 49
> Thread(s) per core: 2
> Core(s) per socket: 64
> Socket(s): 1
> Stepping:0
> Frequency boost: disabled
> CPU max MHz: 4308.3979
> CPU min MHz: 2200.
> BogoMIPS:5390.20
> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush mmx fxsr sse sse2
> ...
> Virtualization features:
> Virtualization:AMD-V
> Caches (sum of all):
> L1d: 2 MiB (64 instances)
> L1i: 2 MiB (64 instances)
> L2:32 MiB (64 instances)
> L3:256 MiB (16 instances)
> NUMA:
> NUMA node(s): 1
> NUMA node0 CPU(s): 0-127
> Vulnerabilities:
> Itlb multihit: Not affected
> L1tf: Not affected
> Mds: Not affected
> Meltdown: Not affected
> Mmio stale data: Not affected
> Retbleed: Mitigation; untrained return thunk; SMT enabled with
> STIBP protection
> Spec store bypass: Mitigation; Speculative Store Bypass disabled via
> prctl
> Spectre v1:Mitigation; usercopy/swapgs barriers and __user
> pointer sanitization
> Spectre v2:Mitigation; Retpolines, IBPB conditional, STIBP
> always-on, RSB filling, PBRSB-eIBRS Not affected
> Srbds: Not affected
> Tsx async abort: Not affected
>
> HOST $ numactl -H
> available: 1 nodes (0)
> node 0 cpus: 0-127
> node 0 size: 257542 MB
> node 0 free: 224855 MB
> node distances:
> node 0
> 0: 10
>
> HOST $ cat /sys/class/nvme/nvme0/model
> INTEL SSDPF21Q800GB
>
> HOST $ cat /sys/class/nvme/nvme0/numa_node
> 0
>
> Software
>
> HOST $ cat /etc/lsb-release
> DISTRIB_ID=Ubuntu
> DISTRIB_RELEASE=22.04
> DISTRIB_CODENAME=jammy
> DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
>
> HOST $ uname -a
> Linux x86 6.4.0-rc5+ #1 SMP PREEMPT_DYNAMIC Wed Jun 7 22:17:47 UTC 2023
> x86_64 x86_64 x86_64 GNU/Linux
>
> HOST $ cat /proc/swaps
> Filename Type Size UsedPriority
> /dev/nvme0n1p2partition4668383560 -2
>
> HOST $ cat /sys/kernel/mm/lru_gen/enabled
> 0x000f
>
> HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled
> always madvise [never]
>
> HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag
> always defer defer+madvise madvise [never]
>
> Procedure
> =
> HOST $ git clone https://github.com/google/multichase
>
> HOST $
> HOST $
>
> HOST $ cp multichase/multichase ./initrd/bin/
> HOST $ sed -i \
> "/^maybe_break top$/i multichase -t 2 -m 4g -n 28800; poweroff" \
I was reminded that I missed one parameter above, i.e.,
"/^maybe_break top$/i multichase -N -t 2 -m 4g -n 28800; poweroff" \
^^
> ./initrd/init
>
> HOST $
>
> HOST $ cat run_microvms.sh
> memcgs=64
>
> run() {
> path=/sys/fs/cgroup/memcg$1
>
> mkdir $path
> echo $BASHPID >$path/cgroup.procs
And one line here:
echo 4000m >$path/memory.min # or the largest size that doesn't cause OOM kills
> qemu-system-x86_64 -M microvm,accel=kvm -cpu host -smp 2 -m 6g \
> -nographic -kernel /boot/vmlinuz -initrd ./initrd.img \
> -append "console=ttyS0 loglevel=0"
> }
>
> for ((memcg = 0; memcg < $memcgs; memcg++)); do
> run $memcg &
> done
>
> wait
>
> Results
> ===
> Before [1]AfterChange
> --
> Total samples6824 7237 +6%
>
> Notes
> =
> [1] "mm: rmap: Don't flush TLB after checking PTE young for page
> reference" was included so that the comparison is apples to
> Apples.
> https://lore.kernel.org/r/20220706112041.3831-1-21cn...@gmail.com/
