Hi Marc,
在 2022/11/30 15:53, Marc Zyngier 写道:
On Wed, 30 Nov 2022 02:52:35 +0000,
"chenxiang (M)" <chenxian...@hisilicon.com> wrote:
Hi,
We boot the VM using following commands (with nvdimm on) (qemu
version 6.1.50, kernel 6.0-r4):
How relevant is the presence of the nvdimm? Do you observe the failure
without this?
We didn't see the failure without it.
qemu-system-aarch64 -machine
virt,kernel_irqchip=on,gic-version=3,nvdimm=on -kernel
/home/kernel/Image -initrd /home/mini-rootfs/rootfs.cpio.gz -bios
/root/QEMU_EFI.FD -cpu host -enable-kvm -net none -nographic -m
2G,maxmem=64G,slots=3 -smp 4 -append 'rdinit=init console=ttyAMA0
ealycon=pl0ll,0x90000000 pcie_ports=native pciehp.pciehp_debug=1'
-object memory-backend-ram,id=ram1,size=10G -device
nvdimm,id=dimm1,memdev=ram1 -device ioh3420,id=root_port1,chassis=1
-device vfio-pci,host=7d:01.0,id=net0,bus=root_port1
Then in VM we insmod a module, vmalloc error occurs as follows (kernel
5.19-rc4 is normal, and the issue is still on kernel 6.1-rc4):
estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
[ 8.186563] vmap allocation for size 20480 failed: use
vmalloc=<size> to increase size
Have you tried increasing the vmalloc size to check that this is
indeed the problem?
[...]
I didn't increase the vmalloc size, but i check the vmall size and i
think it is big enough when the issue occurs:
estuary:/$ insmod /lib/modules/$(uname -r)/hnae3.ko
[ 4.879899] vmap allocation for size 20480 failed: use vmalloc=<size>
to increase size
[ 4.880643] insmod: vmalloc error: size 16384, vm_struct allocation
failed, mode:0xcc0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
[ 4.881802] CPU: 1 PID: 230 Comm: insmod Not tainted 6.1.0-rc4+ #21
[ 4.882414] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0
02/06/2015
[ 4.883082] Call trace:
[ 4.883333] dump_backtrace.part.0+0xc4/0xd0
[ 4.883766] show_stack+0x20/0x50
[ 4.884091] dump_stack_lvl+0x68/0x84
[ 4.884450] dump_stack+0x18/0x34
[ 4.884778] warn_alloc+0x11c/0x1bc
[ 4.885124] __vmalloc_node_range+0x50c/0x64c
[ 4.885553] module_alloc+0xf4/0x100
[ 4.885902] load_module+0x858/0x1e90
[ 4.886265] __do_sys_init_module+0x1c0/0x200
[ 4.886699] __arm64_sys_init_module+0x24/0x30
[ 4.887147] invoke_syscall+0x50/0x120
[ 4.887516] el0_svc_common.constprop.0+0x58/0x190
[ 4.887993] do_el0_svc+0x34/0xc0
[ 4.888327] el0_svc+0x2c/0xb4
[ 4.888631] el0t_64_sync_handler+0xb8/0xbc
[ 4.889046] el0t_64_sync+0x19c/0x1a0
[ 4.889423] Mem-Info:
[ 4.889639] active_anon:9679 inactive_anon:63094 isolated_anon:0
[ 4.889639] active_file:0 inactive_file:0 isolated_file:0
[ 4.889639] unevictable:0 dirty:0 writeback:0
[ 4.889639] slab_reclaimable:3322 slab_unreclaimable:3082
[ 4.889639] mapped:873 shmem:72569 pagetables:34
[ 4.889639] sec_pagetables:0 bounce:0
[ 4.889639] kernel_misc_reclaimable:0
[ 4.889639] free:416212 free_pcp:4414 free_cma:0
[ 4.893362] Node 0 active_anon:38716kB inactive_anon:252376kB
active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB
isolated(file):0kB mapped:3492kB dirty:0kB writeback:0kB shmem:290276kB
shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB
kernel_stack:1904kB pagetables:136kB sec_pagetables:0kB
all_unreclaimable? no
[ 4.896343] Node 0 DMA free:1664848kB boost:0kB min:22528kB
low:28160kB high:33792kB reserved_highatomic:0KB active_anon:38716kB
inactive_anon:252376kB active_file:0kB inactive_file:0kB unevictable:0kB
writepending:0kB present:2097152kB managed:2010376kB mlocked:0kB
bounce:0kB free_pcp:17704kB local_pcp:3668kB free_cma:0kB
[ 4.899097] lowmem_reserve[]: 0 0 0 0 0
[ 4.899466] Node 0 DMA: 2*4kB (UM) 1*8kB (M) 2*16kB (UM) 1*32kB (M)
2*64kB (ME) 1*128kB (U) 2*256kB (ME) 2*512kB (M) 6*1024kB (UME) 5*2048kB
(UM) 402*4096kB (M) = 1664848kB
[ 4.900865] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[ 4.901648] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=32768kB
[ 4.902526] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[ 4.903354] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=64kB
[ 4.904173] 72569 total pagecache pages
[ 4.904524] 0 pages in swap cache
[ 4.904831] Free swap = 0kB
[ 4.905109] Total swap = 0kB
[ 4.905407] 524288 pages RAM
[ 4.905696] 0 pages HighMem/MovableOnly
[ 4.906085] 21694 pages reserved
[ 4.906388] 0 pages hwpoisoned
insmod: can't insert '/lib/modules/6.1.0-rc4+/hnae3.ko': Cannot allocate
memory
estuary:/$ insmod /lib/modules/$(uname -r)/hns3.ko
[ 4.911599] vmap allocation for size 122880 failed: use
vmalloc=<size> to increase size
insmod: can't insert '/lib/modules/6.1.0-rc4+/hns3.ko': Cannot allocate
memory
estuary:/$ insmod /lib/modules/$(uname -r)/hclge.ko
[ 4.917761] vmap allocation for size 319488 failed: use
vmalloc=<size> to increase size
insmod: can't insert '/lib/modules/6.1.0-rc4+/hclge.ko': Cannot allocate
memory
estuary:/$ insmod /lib/modules/$(uname -r)/hclgevf.ko
[ 5.160299] vmap allocation for size 73728 failed: use vmalloc=<size>
to increase size
insmod: can't insert '/lib/modules/6.1.0-rc4+/hclgevf.ko': Cannot
allocate memory
estuary:/$
estuary:/$ cat /proc/meminfo
MemTotal: 2010376 kB
MemFree: 1664848 kB
MemAvailable: 1637744 kB
Buffers: 0 kB
Cached: 290276 kB
SwapCached: 0 kB
Active: 39456 kB
Inactive: 251628 kB
Active(anon): 39456 kB
Inactive(anon): 251628 kB
Active(file): 0 kB
Inactive(file): 0 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 860 kB
Mapped: 3368 kB
Shmem: 290276 kB
KReclaimable: 13372 kB
Slab: 25732 kB
SReclaimable: 13372 kB
SUnreclaim: 12360 kB
KernelStack: 1872 kB
PageTables: 136 kB
SecPageTables: 0 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 1005188 kB
Committed_AS: 292076 kB
VmallocTotal: 133143592960 kB
VmallocUsed: 2612 kB
VmallocChunk: 0 kB
Percpu: 672 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
estuary:/$
We git bisect the code, and find the patch c5a89f75d2a ("arm64: kaslr:
defer initialization to initcall where permitted").
I guess you mean commit fc5a89f75d2a instead, right?
Right
Do you have any idea about the issue?
I sort of suspect that the nvdimm gets vmap-ed and consumes a large
portion of the vmalloc space, but you give very little information
that could help here...
M.
