Hello, I'm currently working on re-enabling our daily OVN tests in Incus (the LXD fork).
Unfortunately I'm not having much luck getting our testsuite to go all the way through as it's triggering kernel panics. Here is the stack trace I'm getting: ``` [ 484.818312] BUG: TASK stack guard page was hit at 0000000075451289 (stack is 00000000d168f1fe..00000000c11d6033) [ 484.818331] stack guard page: 0000 [#1] PREEMPT SMP PTI [ 484.818338] CPU: 0 PID: 19480 Comm: revalidator49 Not tainted 6.5.9-zabbly+ #ubuntu22.04 [ 484.818343] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS unknown 2/2/2022 [ 484.818345] RIP: 0010:reserve_sfa_size+0x5/0x110 [openvswitch] [ 484.818412] Code: cc 0f 0b eb d1 48 c7 c0 f4 ff ff ff 5d c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <55> 48 89 e5 41 57 41 89 d7 41 56 44 8d 76 03 41 55 41 83 e6 fc 41 [ 484.818414] RSP: 0018:ffffa8a840df8000 EFLAGS: 00010246 [ 484.818417] RAX: ffff9411b4730000 RBX: 0000000000000000 RCX: 0000000000000000 [ 484.818419] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffffa8a840dfb278 [ 484.818420] RBP: ffffa8a840df8030 R08: 0000000000000001 R09: 000000000000dd86 [ 484.818421] R10: ffffa8a840df80ff R11: ffffa8a840dfb280 R12: ffffa8a84113247c [ 484.818422] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000019 [ 484.818424] FS: 00007f25032f5640(0000) GS:ffff941237200000(0000) knlGS:0000000000000000 [ 484.818426] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 484.818428] CR2: ffffa8a840df7ff8 CR3: 000000011315a001 CR4: 0000000000170ef0 [ 484.818433] Call Trace: [ 484.818445] <#DF> [ 484.818450] ? show_regs+0x6a/0x80 [ 484.818467] ? die+0x38/0xa0 [ 484.818470] ? handle_stack_overflow+0x4e/0x60 [ 484.818478] ? exc_double_fault+0x106/0x190 [ 484.818502] ? asm_exc_double_fault+0x1f/0x30 [ 484.818518] ? reserve_sfa_size+0x5/0x110 [openvswitch] [ 484.818530] </#DF> [ 484.818531] <TASK> [ 484.818531] ? __add_action+0x2a/0x80 [openvswitch] [ 484.818542] validate_set.constprop.0+0x23e/0x410 [openvswitch] [ 484.818555] __ovs_nla_copy_actions+0x40f/0xfb0 [openvswitch] [ 484.818566] ? reserve_sfa_size+0x32/0x110 [openvswitch] [ 484.818577] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818593] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818604] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818615] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818630] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818642] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818671] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818683] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818694] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818705] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818717] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818728] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818739] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818750] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818761] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818772] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818783] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818794] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818806] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818817] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818828] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818839] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818850] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818861] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818872] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818883] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818894] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818905] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818916] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818927] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818939] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818950] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818961] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818972] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818983] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.818994] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819005] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819016] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819027] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819038] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819049] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819060] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819071] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819082] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819093] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819104] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819115] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819126] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819137] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819148] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819165] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819176] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819187] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819198] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819267] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819279] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819290] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819301] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819312] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.819323] __ovs_nla_copy_actions+0xb4e/0xfb0 [openvswitch] [ 484.819334] ? __kmalloc+0xca/0x140 [ 484.819345] ovs_nla_copy_actions+0x98/0xf0 [openvswitch] [ 484.819355] get_flow_actions+0x60/0xc0 [openvswitch] [ 484.819369] ovs_nla_init_match_and_action.constprop.0+0xb5/0x120 [openvswitch] [ 484.819383] ovs_flow_cmd_set+0xb3/0x3f0 [openvswitch] [ 484.819401] genl_family_rcv_msg_doit.isra.0+0xcc/0x120 [ 484.819422] genl_rcv_msg+0x17d/0x2b0 [ 484.819425] ? __pfx_ovs_flow_cmd_set+0x10/0x10 [openvswitch] [ 484.819435] ? __pfx_genl_rcv_msg+0x10/0x10 [ 484.819438] netlink_rcv_skb+0x5c/0x110 [ 484.819443] genl_rcv+0x29/0x40 [ 484.819445] netlink_unicast+0x23f/0x350 [ 484.819448] netlink_sendmsg+0x25d/0x4f0 [ 484.819452] ____sys_sendmsg+0x3a5/0x3c0 [ 484.819461] ___sys_sendmsg+0x88/0xd0 [ 484.819465] ? copy_msghdr_from_user+0x71/0xb0 [ 484.819469] ? ___sys_recvmsg+0x88/0xd0 [ 484.819472] ? mlock_drain_local+0x38/0x50 [ 484.819483] ? __fget_light+0xad/0x140 [ 484.819490] __sys_sendmsg+0x65/0xc0 [ 484.819495] __x64_sys_sendmsg+0x1d/0x30 [ 484.819498] do_syscall_64+0x5c/0x90 [ 484.819501] ? __x64_sys_recvmsg+0x1d/0x30 [ 484.819503] ? do_syscall_64+0x69/0x90 [ 484.819505] ? exit_to_user_mode_prepare+0x35/0x180 [ 484.819512] ? syscall_exit_to_user_mode+0x34/0x50 [ 484.819522] ? __x64_sys_mremap+0x25/0x30 [ 484.819530] ? do_syscall_64+0x69/0x90 [ 484.819532] ? exit_to_user_mode_prepare+0x35/0x180 [ 484.819535] ? syscall_exit_to_user_mode+0x34/0x50 [ 484.819538] ? __x64_sys_mremap+0x25/0x30 [ 484.819541] ? do_syscall_64+0x69/0x90 [ 484.819543] ? do_syscall_64+0x69/0x90 [ 484.819545] ? do_syscall_64+0x69/0x90 [ 484.819547] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 484.819550] RIP: 0033:0x7f2503d27b4d [ 484.819553] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 3a 8f f6 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 7e 8f f6 ff 48 [ 484.819558] RSP: 002b:00007f25032d36d0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e [ 484.819561] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2503d27b4d [ 484.819563] RDX: 0000000000000000 RSI: 00007f25032d3760 RDI: 0000000000000015 [ 484.819564] RBP: 00007f25032d4550 R08: 0000000000000000 R09: 0000000000000188 [ 484.819566] R10: 000000000000000b R11: 0000000000000293 R12: 000055768c207140 [ 484.819567] R13: 0000000000000001 R14: 00000000000011ef R15: 00007f25032d3760 [ 484.819570] </TASK> [ 484.819571] Modules linked in: dummy veth nf_conntrack_netlink nft_masq nft_chain_nat nf_tables vhost_vsock vhost vhost_iotlb nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 tls bridge stp llc binfmt_misc intel_rapl_msr intel_rapl_common kvm_intel nls_iso8859_1 kvm irqbypass rapl input_leds joydev serio_raw 9pnet_virtio vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common 9pnet virtio_input vmgenid mac_hid qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vsock ramoops reed_solomon pstore_blk pstore_zone virtiofs efi_pstore virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel virtio_gpu virtio_dma_buf drm_shmem_helper sha512_ssse3 aesni_intel crypto_simd drm_kms_helper cryptd drm i2c_i801 psmouse xhci_pci [ 484.819720] i2c_smbus virtio_net ahci libahci net_failover lpc_ich failover xhci_pci_renesas [ 484.819736] ---[ end trace 0000000000000000 ]--- [ 484.821609] ------------[ cut here ]------------ [ 484.821611] WARNING: CPU: 0 PID: 19480 at arch/x86/kernel/fpu/core.c:60 irq_fpu_usable+0x3e/0x50 [ 484.821625] Modules linked in: dummy veth nf_conntrack_netlink nft_masq nft_chain_nat nf_tables vhost_vsock vhost vhost_iotlb nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 tls bridge stp llc binfmt_misc intel_rapl_msr intel_rapl_common kvm_intel nls_iso8859_1 kvm irqbypass rapl input_leds joydev serio_raw 9pnet_virtio vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common 9pnet virtio_input vmgenid mac_hid qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vsock ramoops reed_solomon pstore_blk pstore_zone virtiofs efi_pstore virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel virtio_gpu virtio_dma_buf drm_shmem_helper sha512_ssse3 aesni_intel crypto_simd drm_kms_helper cryptd drm i2c_i801 psmouse xhci_pci [ 484.821672] i2c_smbus virtio_net ahci libahci net_failover lpc_ich failover xhci_pci_renesas [ 484.821677] CPU: 0 PID: 19480 Comm: revalidator49 Tainted: G D 6.5.9-zabbly+ #ubuntu22.04 [ 484.821680] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS unknown 2/2/2022 [ 484.821681] RIP: 0010:irq_fpu_usable+0x3e/0x50 [ 484.821685] Code: f0 00 75 26 65 8a 0d 19 62 dc 6b 31 c0 84 c9 75 13 b8 01 00 00 00 f7 c2 00 00 0f 00 74 06 80 e6 ff 0f 94 c0 5d c3 cc cc cc cc <0f> 0b 31 c0 5d c3 cc cc cc cc 0f 1f 84 00 00 00 00 00 90 90 90 90 [ 484.821687] RSP: 0018:fffffe76ae82bae8 EFLAGS: 00010006 [ 484.821689] RAX: 0000000000000000 RBX: 0000000000000003 RCX: ffff9410c0253000 [ 484.821691] RDX: 0000000080110002 RSI: ffff9410c0253fe0 RDI: 0000000000000003 [ 484.821692] RBP: fffffe76ae82bae8 R08: 000000027fee4067 R09: 0000000000000000 [ 484.821693] R10: 0000000000000000 R11: 0000000139cdc067 R12: 8000000000000006 [ 484.821695] R13: 0000000000000007 R14: fffffe76ae82bb68 R15: fffffe76ae82bb70 [ 484.821696] FS: 00007f25032f5640(0000) GS:ffff941237200000(0000) knlGS:0000000000000000 [ 484.821698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 484.821699] CR2: ffffa8a840df7ff8 CR3: 000000011315a001 CR4: 0000000000170ef0 [ 484.821703] Call Trace: [ 484.821705] <#DF> [ 484.821707] ? show_regs+0x6a/0x80 [ 484.821710] ? irq_fpu_usable+0x3e/0x50 [ 484.821713] ? __warn+0x8d/0x150 [ 484.821722] ? irq_fpu_usable+0x3e/0x50 [ 484.821725] ? report_bug+0x199/0x1b0 [ 484.821745] ? handle_bug+0x46/0x80 [ 484.821749] ? exc_invalid_op+0x19/0x70 [ 484.821752] ? asm_exc_invalid_op+0x1b/0x20 [ 484.821757] ? irq_fpu_usable+0x3e/0x50 [ 484.821760] kernel_fpu_begin_mask+0x2b/0xc0 [ 484.821765] virt_efi_query_variable_info_nonblocking+0x66/0x140 [ 484.822148] efi_query_variable_store+0x178/0x1c0 [ 484.822165] efivar_set_variable_locked+0xad/0xf0 [ 484.822171] efi_pstore_write+0x152/0x1d0 [efi_pstore] [ 484.822179] pstore_dump+0x242/0x430 [ 484.822188] ? __pfx_efi_pstore_write+0x10/0x10 [efi_pstore] [ 484.822193] ? pstore_dump+0x242/0x430 [ 484.822201] kmsg_dump+0x63/0x80 [ 484.822208] oops_exit+0x30/0x40 [ 484.822211] oops_end+0x6b/0xf0 [ 484.822214] die+0x83/0xa0 [ 484.822217] handle_stack_overflow+0x4e/0x60 [ 484.822220] exc_double_fault+0x106/0x190 [ 484.822224] asm_exc_double_fault+0x1f/0x30 [ 484.822228] RIP: 0010:reserve_sfa_size+0x5/0x110 [openvswitch] [ 484.822242] Code: cc 0f 0b eb d1 48 c7 c0 f4 ff ff ff 5d c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <55> 48 89 e5 41 57 41 89 d7 41 56 44 8d 76 03 41 55 41 83 e6 fc 41 [ 484.822244] RSP: 0018:ffffa8a840df8000 EFLAGS: 00010246 [ 484.822247] RAX: ffff9411b4730000 RBX: 0000000000000000 RCX: 0000000000000000 [ 484.822248] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffffa8a840dfb278 [ 484.822249] RBP: ffffa8a840df8030 R08: 0000000000000001 R09: 000000000000dd86 [ 484.822250] R10: ffffa8a840df80ff R11: ffffa8a840dfb280 R12: ffffa8a84113247c [ 484.822251] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000019 [ 484.822254] </#DF> [ 484.822255] <TASK> [ 484.822256] ? __add_action+0x2a/0x80 [openvswitch] [ 484.822267] validate_set.constprop.0+0x23e/0x410 [openvswitch] [ 484.822279] __ovs_nla_copy_actions+0x40f/0xfb0 [openvswitch] [ 484.822290] ? reserve_sfa_size+0x32/0x110 [openvswitch] [ 484.822302] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822314] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822325] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822336] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822348] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822359] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822370] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822385] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822397] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822408] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822419] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822430] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822441] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822452] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822463] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822474] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822486] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822497] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822508] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822519] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822530] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822541] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822552] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822564] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822575] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822587] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822601] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822613] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822624] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822635] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822646] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822657] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822668] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822696] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822707] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822719] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822730] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822741] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822752] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822763] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822774] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822785] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822796] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822807] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822819] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822830] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822841] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822890] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822902] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822913] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822924] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822935] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822946] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822957] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822968] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822979] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.822990] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823017] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823028] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823039] __ovs_nla_copy_actions+0xb4e/0xfb0 [openvswitch] [ 484.823050] ? __kmalloc+0xca/0x140 [ 484.823053] ovs_nla_copy_actions+0x98/0xf0 [openvswitch] [ 484.823063] get_flow_actions+0x60/0xc0 [openvswitch] [ 484.823077] ovs_nla_init_match_and_action.constprop.0+0xb5/0x120 [openvswitch] [ 484.823091] ovs_flow_cmd_set+0xb3/0x3f0 [openvswitch] [ 484.823106] genl_family_rcv_msg_doit.isra.0+0xcc/0x120 [ 484.823111] genl_rcv_msg+0x17d/0x2b0 [ 484.823114] ? __pfx_ovs_flow_cmd_set+0x10/0x10 [openvswitch] [ 484.823124] ? __pfx_genl_rcv_msg+0x10/0x10 [ 484.823127] netlink_rcv_skb+0x5c/0x110 [ 484.823131] genl_rcv+0x29/0x40 [ 484.823134] netlink_unicast+0x23f/0x350 [ 484.823137] netlink_sendmsg+0x25d/0x4f0 [ 484.823151] ____sys_sendmsg+0x3a5/0x3c0 [ 484.823190] ___sys_sendmsg+0x88/0xd0 [ 484.823194] ? copy_msghdr_from_user+0x71/0xb0 [ 484.823198] ? ___sys_recvmsg+0x88/0xd0 [ 484.823202] ? mlock_drain_local+0x38/0x50 [ 484.823206] ? __fget_light+0xad/0x140 [ 484.823209] __sys_sendmsg+0x65/0xc0 [ 484.823213] __x64_sys_sendmsg+0x1d/0x30 [ 484.823217] do_syscall_64+0x5c/0x90 [ 484.823220] ? __x64_sys_recvmsg+0x1d/0x30 [ 484.823222] ? do_syscall_64+0x69/0x90 [ 484.823224] ? exit_to_user_mode_prepare+0x35/0x180 [ 484.823226] ? syscall_exit_to_user_mode+0x34/0x50 [ 484.823230] ? __x64_sys_mremap+0x25/0x30 [ 484.823233] ? do_syscall_64+0x69/0x90 [ 484.823235] ? exit_to_user_mode_prepare+0x35/0x180 [ 484.823237] ? syscall_exit_to_user_mode+0x34/0x50 [ 484.823240] ? __x64_sys_mremap+0x25/0x30 [ 484.823243] ? do_syscall_64+0x69/0x90 [ 484.823245] ? do_syscall_64+0x69/0x90 [ 484.823247] ? do_syscall_64+0x69/0x90 [ 484.823250] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 484.823253] RIP: 0033:0x7f2503d27b4d [ 484.823256] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 3a 8f f6 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 7e 8f f6 ff 48 [ 484.823258] RSP: 002b:00007f25032d36d0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e [ 484.823261] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2503d27b4d [ 484.823263] RDX: 0000000000000000 RSI: 00007f25032d3760 RDI: 0000000000000015 [ 484.823264] RBP: 00007f25032d4550 R08: 0000000000000000 R09: 0000000000000188 [ 484.823265] R10: 000000000000000b R11: 0000000000000293 R12: 000055768c207140 [ 484.823266] R13: 0000000000000001 R14: 00000000000011ef R15: 00007f25032d3760 [ 484.823269] </TASK> [ 484.823270] ---[ end trace 0000000000000000 ]--- [ 484.823281] ------------[ cut here ]------------ [ 484.823282] WARNING: CPU: 0 PID: 19480 at arch/x86/kernel/fpu/core.c:425 kernel_fpu_begin_mask+0xad/0xc0 [ 484.823287] Modules linked in: dummy veth nf_conntrack_netlink nft_masq nft_chain_nat nf_tables vhost_vsock vhost vhost_iotlb nfnetlink_cttimeout nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 tls bridge stp llc binfmt_misc intel_rapl_msr intel_rapl_common kvm_intel nls_iso8859_1 kvm irqbypass rapl input_leds joydev serio_raw 9pnet_virtio vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common 9pnet virtio_input vmgenid mac_hid qemu_fw_cfg sch_fq_codel dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vsock ramoops reed_solomon pstore_blk pstore_zone virtiofs efi_pstore virtio_rng ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel virtio_gpu virtio_dma_buf drm_shmem_helper sha512_ssse3 aesni_intel crypto_simd drm_kms_helper cryptd drm i2c_i801 psmouse xhci_pci [ 484.823329] i2c_smbus virtio_net ahci libahci net_failover lpc_ich failover xhci_pci_renesas [ 484.823334] CPU: 0 PID: 19480 Comm: revalidator49 Tainted: G D W 6.5.9-zabbly+ #ubuntu22.04 [ 484.823336] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009)/LXD, BIOS unknown 2/2/2022 [ 484.823338] RIP: 0010:kernel_fpu_begin_mask+0xad/0xc0 [ 484.823342] Code: 34 48 8b 5d f8 c9 c3 cc cc cc cc 48 8b 07 f6 c4 40 75 be f0 80 4f 01 40 48 81 c7 c0 15 00 00 e8 a9 fe ff ff eb ab db e3 eb c8 <0f> 0b e9 7b ff ff ff 0f 0b eb 82 e8 c3 22 fa 00 0f 1f 00 90 90 90 [ 484.823344] RSP: 0018:fffffe76ae82baf8 EFLAGS: 00010046 [ 484.823346] RAX: 0000000000000000 RBX: 0000000000000003 RCX: ffff9410c0253000 [ 484.823348] RDX: 0000000080110002 RSI: ffff9410c0253fe0 RDI: 0000000000000003 [ 484.823349] RBP: fffffe76ae82bb10 R08: 000000027fee4067 R09: 0000000000000000 [ 484.823350] R10: 0000000000000000 R11: 0000000139cdc067 R12: 8000000000000006 [ 484.823351] R13: 0000000000000007 R14: fffffe76ae82bb68 R15: fffffe76ae82bb70 [ 484.823357] FS: 00007f25032f5640(0000) GS:ffff941237200000(0000) knlGS:0000000000000000 [ 484.823359] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 484.823361] CR2: ffffa8a840df7ff8 CR3: 000000011315a001 CR4: 0000000000170ef0 [ 484.823368] Call Trace: [ 484.823369] <#DF> [ 484.823370] ? show_regs+0x6a/0x80 [ 484.823373] ? kernel_fpu_begin_mask+0xad/0xc0 [ 484.823377] ? __warn+0x8d/0x150 [ 484.823380] ? kernel_fpu_begin_mask+0xad/0xc0 [ 484.823384] ? report_bug+0x199/0x1b0 [ 484.823388] ? handle_bug+0x46/0x80 [ 484.823391] ? exc_invalid_op+0x19/0x70 [ 484.823395] ? asm_exc_invalid_op+0x1b/0x20 [ 484.823399] ? kernel_fpu_begin_mask+0xad/0xc0 [ 484.823402] ? kernel_fpu_begin_mask+0x2b/0xc0 [ 484.823406] virt_efi_query_variable_info_nonblocking+0x66/0x140 [ 484.823410] efi_query_variable_store+0x178/0x1c0 [ 484.823414] efivar_set_variable_locked+0xad/0xf0 [ 484.823418] efi_pstore_write+0x152/0x1d0 [efi_pstore] [ 484.823425] pstore_dump+0x242/0x430 [ 484.823428] ? __pfx_efi_pstore_write+0x10/0x10 [efi_pstore] [ 484.823433] ? pstore_dump+0x242/0x430 [ 484.823437] kmsg_dump+0x63/0x80 [ 484.823440] oops_exit+0x30/0x40 [ 484.823442] oops_end+0x6b/0xf0 [ 484.823445] die+0x83/0xa0 [ 484.823448] handle_stack_overflow+0x4e/0x60 [ 484.823450] exc_double_fault+0x106/0x190 [ 484.823454] asm_exc_double_fault+0x1f/0x30 [ 484.823457] RIP: 0010:reserve_sfa_size+0x5/0x110 [openvswitch] [ 484.823468] Code: cc 0f 0b eb d1 48 c7 c0 f4 ff ff ff 5d c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <55> 48 89 e5 41 57 41 89 d7 41 56 44 8d 76 03 41 55 41 83 e6 fc 41 [ 484.823470] RSP: 0018:ffffa8a840df8000 EFLAGS: 00010246 [ 484.823472] RAX: ffff9411b4730000 RBX: 0000000000000000 RCX: 0000000000000000 [ 484.823473] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffffa8a840dfb278 [ 484.823474] RBP: ffffa8a840df8030 R08: 0000000000000001 R09: 000000000000dd86 [ 484.823476] R10: ffffa8a840df80ff R11: ffffa8a840dfb280 R12: ffffa8a84113247c [ 484.823477] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000019 [ 484.823480] </#DF> [ 484.823481] <TASK> [ 484.823482] ? __add_action+0x2a/0x80 [openvswitch] [ 484.823492] validate_set.constprop.0+0x23e/0x410 [openvswitch] [ 484.823503] __ovs_nla_copy_actions+0x40f/0xfb0 [openvswitch] [ 484.823514] ? reserve_sfa_size+0x32/0x110 [openvswitch] [ 484.823524] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823535] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823546] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823558] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823574] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823585] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823597] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823608] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823620] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823631] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823642] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823653] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823664] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823675] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823686] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823697] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823709] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823720] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823731] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823742] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823753] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823764] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823775] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823786] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823797] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823809] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823820] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823831] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823842] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823853] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823864] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823879] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823891] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823902] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823913] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823924] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823935] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823946] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823957] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823968] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823979] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.823990] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824001] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824012] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824023] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824035] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824046] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824057] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824068] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824079] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824094] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824105] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824116] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824127] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824138] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824149] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824160] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824199] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824210] __ovs_nla_copy_actions+0xbda/0xfb0 [openvswitch] [ 484.824221] __ovs_nla_copy_actions+0xb4e/0xfb0 [openvswitch] [ 484.824231] ? __kmalloc+0xca/0x140 [ 484.824234] ovs_nla_copy_actions+0x98/0xf0 [openvswitch] [ 484.824244] get_flow_actions+0x60/0xc0 [openvswitch] [ 484.824258] ovs_nla_init_match_and_action.constprop.0+0xb5/0x120 [openvswitch] [ 484.824272] ovs_flow_cmd_set+0xb3/0x3f0 [openvswitch] [ 484.824286] genl_family_rcv_msg_doit.isra.0+0xcc/0x120 [ 484.824291] genl_rcv_msg+0x17d/0x2b0 [ 484.824294] ? __pfx_ovs_flow_cmd_set+0x10/0x10 [openvswitch] [ 484.824304] ? __pfx_genl_rcv_msg+0x10/0x10 [ 484.824307] netlink_rcv_skb+0x5c/0x110 [ 484.824311] genl_rcv+0x29/0x40 [ 484.824314] netlink_unicast+0x23f/0x350 [ 484.824317] netlink_sendmsg+0x25d/0x4f0 [ 484.824321] ____sys_sendmsg+0x3a5/0x3c0 [ 484.824324] ___sys_sendmsg+0x88/0xd0 [ 484.824328] ? copy_msghdr_from_user+0x71/0xb0 [ 484.824332] ? ___sys_recvmsg+0x88/0xd0 [ 484.824335] ? mlock_drain_local+0x38/0x50 [ 484.824338] ? __fget_light+0xad/0x140 [ 484.824341] __sys_sendmsg+0x65/0xc0 [ 484.824345] __x64_sys_sendmsg+0x1d/0x30 [ 484.824355] do_syscall_64+0x5c/0x90 [ 484.824358] ? __x64_sys_recvmsg+0x1d/0x30 [ 484.824359] ? do_syscall_64+0x69/0x90 [ 484.824362] ? exit_to_user_mode_prepare+0x35/0x180 [ 484.824365] ? syscall_exit_to_user_mode+0x34/0x50 [ 484.824367] ? __x64_sys_mremap+0x25/0x30 [ 484.824372] ? do_syscall_64+0x69/0x90 [ 484.824374] ? exit_to_user_mode_prepare+0x35/0x180 [ 484.824376] ? syscall_exit_to_user_mode+0x34/0x50 [ 484.824379] ? __x64_sys_mremap+0x25/0x30 [ 484.824382] ? do_syscall_64+0x69/0x90 [ 484.824384] ? do_syscall_64+0x69/0x90 [ 484.824386] ? do_syscall_64+0x69/0x90 [ 484.824388] entry_SYSCALL_64_after_hwframe+0x6e/0xd8 [ 484.824391] RIP: 0033:0x7f2503d27b4d [ 484.824393] Code: 28 89 54 24 1c 48 89 74 24 10 89 7c 24 08 e8 3a 8f f6 ff 8b 54 24 1c 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 2e 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 33 44 89 c7 48 89 44 24 08 e8 7e 8f f6 ff 48 [ 484.824397] RSP: 002b:00007f25032d36d0 EFLAGS: 00000293 ORIG_RAX: 000000000000002e [ 484.824400] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f2503d27b4d [ 484.824401] RDX: 0000000000000000 RSI: 00007f25032d3760 RDI: 0000000000000015 [ 484.824402] RBP: 00007f25032d4550 R08: 0000000000000000 R09: 0000000000000188 [ 484.824403] R10: 000000000000000b R11: 0000000000000293 R12: 000055768c207140 [ 484.824407] R13: 0000000000000001 R14: 00000000000011ef R15: 00007f25032d3760 [ 484.824411] </TASK> [ 484.824412] ---[ end trace 0000000000000000 ]--- [ 486.680456] RIP: 0010:reserve_sfa_size+0x5/0x110 [openvswitch] [ 486.680605] Code: cc 0f 0b eb d1 48 c7 c0 f4 ff ff ff 5d c3 cc cc cc cc 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 <55> 48 89 e5 41 57 41 89 d7 41 56 44 8d 76 03 41 55 41 83 e6 fc 41 [ 486.680609] RSP: 0018:ffffa8a840df8000 EFLAGS: 00010246 [ 486.680616] RAX: ffff9411b4730000 RBX: 0000000000000000 RCX: 0000000000000000 [ 486.680620] RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffffa8a840dfb278 [ 486.680622] RBP: ffffa8a840df8030 R08: 0000000000000001 R09: 000000000000dd86 [ 486.680624] R10: ffffa8a840df80ff R11: ffffa8a840dfb280 R12: ffffa8a84113247c [ 486.680625] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000019 [ 486.680627] FS: 00007f25032f5640(0000) GS:ffff941237200000(0000) knlGS:0000000000000000 [ 486.680629] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 486.680630] CR2: ffffa8a840df7ff8 CR3: 000000011315a001 CR4: 0000000000170ef0 [ 486.680646] Kernel panic - not syncing: Fatal exception in interrupt [ 486.680892] Kernel Offset: 0x13200000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 489.122110] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]--- ``` That kernel build is effectively a clean 6.5.9 kernel. The action immediately preceding the kernel panic is the instance being forcefully stopped, making the last command to run prior to panic be `ip link del vethXYZ`. I can reproduce this panic very consistently, though can't easily isolate the particular configuration needed in order for this to trigger. For example, once the machine is rebooted after the panic, I can start/stop those instances at will, without any kernel panic. Stéphane _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev