Hello,
On Tue, Jul 20, 2021 at 10:32:39PM +0000, Andy Smith wrote:
> I have a Debian 10 (buster/stable) dom0 running hypervisor 4.14.2.
> For almost 2 years it's been using the packaged Debian stable kernel
> which is 4.19.x.
>
> Last night I upgraded the kernel to the buster-backports package
> which is based on 5.10.40 and about 4 hours later got this:
>
> Jul 20 02:17:54 lamb kernel: [21061.388607] sg[0]
> phys_addr:0x00000015eb803000 offset:0 length:4096
> dma_address:0x000000209e7b7000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.389775] sg[1]
> phys_addr:0x00000015eb7bc000 offset:0 length:4096
> dma_address:0x000000209e7b8000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.390874] sg[2]
> phys_addr:0x00000015eb809000 offset:0 length:4096
> dma_address:0x000000209e7b9000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.391974] sg[3]
> phys_addr:0x00000015eb766000 offset:0 length:4096
> dma_address:0x000000209e7ba000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.393042] sg[4]
> phys_addr:0x00000015eb7a3000 offset:0 length:4096
> dma_address:0x000000209e7bb000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.394086] sg[5]
> phys_addr:0x00000015eb7c6000 offset:0 length:4096
> dma_address:0x000000209e7bc000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.395078] sg[6]
> phys_addr:0x00000015eb7c2000 offset:0 length:4096
> dma_address:0x000000209e7bd000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.396042] sg[7]
> phys_addr:0x00000015eb7a9000 offset:0 length:4096
> dma_address:0x000000209e7be000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.397004] sg[8]
> phys_addr:0x00000015eb775000 offset:0 length:4096
> dma_address:0x000000209e7bf000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.397971] sg[9]
> phys_addr:0x00000015eb7c7000 offset:0 length:4096
> dma_address:0x00000020ff520000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.398889] sg[10]
> phys_addr:0x00000015eb7cb000 offset:0 length:4096
> dma_address:0x00000020ff521000 dma_length:4096
> Jul 20 02:17:54 lamb kernel: [21061.399814] sg[11]
> phys_addr:0x00000015eb7e3000 offset:0 length:61952
> dma_address:0x00000020ff522000 dma_length:61952
> Jul 20 02:17:54 lamb kernel: [21061.400754] sg[12]
> phys_addr:0x00000015eb7f2200 offset:512 length:24064
> dma_address:0x00000020ff531200 dma_length:24064
> Jul 20 02:17:54 lamb kernel: [21061.401781] ------------[ cut here
> ]------------
> Jul 20 02:17:54 lamb kernel: [21061.402738] Invalid SGL for payload:131072
> nents:13
> Jul 20 02:17:54 lamb kernel: [21061.403724] WARNING: CPU: 1 PID: 12669 at
> drivers/nvme/host/pci.c:716 nvme_map_data+0x7e0/0x820 [nvme]
> Jul 20 02:17:54 lamb kernel: [21061.404728] Modules linked in: binfmt_misc
> ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpmss nf_log_ipv6
> nf_log_ipv4 nf_log_common xt_LOG xt_limit nfnetlink_log nfnetlink xt_NFLOG
> xt_multiport xt_tcpudp ip6table_filter ip6_tables iptable_filter bonding
> btrfs blake2b_generic dm_snapshot dm_bufio intel_rapl_msr intel_rapl_common
> skx_edac nfit libnvdimm intel_powerclamp crc32_pclmul ghash_clmulni_intel
> ipmi_ssif aesni_intel libaes crypto_simd cryptd glue_helper snd_hda_intel
> snd_intel_dspcfg mei_wdt soundwire_intel soundwire_generic_allocation nvme
> wdat_wdt snd_soc_core ast snd_compress watchdog drm_vram_helper
> drm_ttm_helper soundwire_cadence pcspkr nvme_core ttm snd_hda_codec
> drm_kms_helper snd_hda_core i2c_i801 snd_hwdep i2c_smbus cec soundwire_bus
> snd_pcm drm snd_timer snd soundcore igb ptp pps_core i2c_algo_bit joydev
> mei_me sg mei intel_lpss_pci intel_lpss idma64 acpi_ipmi ipmi_si ipmi_devintf
> ioatdma dca wmi ipmi_msghandler button dm_mod xenfs xen_acpi_processor
> Jul 20 02:17:54 lamb kernel: [21061.404831] xen_privcmd xen_pciback
> xen_netback xen_blkback xen_gntalloc xen_gntdev xen_evtchn ip_tables x_tables
> autofs4 ext4 crc16 mbcache jbd2 raid456 libcrc32c crc32c_generic
> async_raid6_recov async_memcpy async_pq async_xor xor async_tx evdev
> hid_generic usbhid hid raid6_pq raid0 multipath linear raid10 raid1 md_mod
> sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common
> xhci_pci ahci libahci crc32c_intel xhci_hcd libata usbcore scsi_mod usb_common
> Jul 20 02:17:54 lamb kernel: [21061.417998] CPU: 1 PID: 12669 Comm: 62.xvda-0
> Not tainted 5.10.0-0.bpo.7-amd64 #1 Debian 5.10.40-1~bpo10+1
> Jul 20 02:17:54 lamb kernel: [21061.418459] Hardware name: Supermicro Super
> Server/X11SRM-VF, BIOS 1.2a 02/18/2019
> Jul 20 02:17:54 lamb kernel: [21061.418922] RIP:
> e030:nvme_map_data+0x7e0/0x820 [nvme]
> Jul 20 02:17:54 lamb kernel: [21061.419354] Code: d0 7b c0 48 c7 c7 40 d6 7b
> c0 e8 5b 44 c9 c0 8b 93 4c 01 00 00 f6 43 1e 04 75 36 8b 73 28 48 c7 c7 20 9c
> 7b c0 e8 8b 71 09 c1 <0f> 0b 41 bd 0a 00 00 00 e9 f7 fe ff ff 48 8d bd 68 02
> 00 00 48 89
> Jul 20 02:17:54 lamb kernel: [21061.420271] RSP: e02b:ffffc90044797930
> EFLAGS: 00010286
> Jul 20 02:17:54 lamb kernel: [21061.420727] RAX: 0000000000000000 RBX:
> ffff888157db4200 RCX: 0000000000000027
> Jul 20 02:17:54 lamb kernel: [21061.421186] RDX: 0000000000000027 RSI:
> ffff888292858a00 RDI: ffff888292858a08
> Jul 20 02:17:54 lamb kernel: [21061.421639] RBP: ffff888103243000 R08:
> 0000000000000000 R09: c00000010000118b
> Jul 20 02:17:54 lamb kernel: [21061.422090] R10: 0000000000165920 R11:
> ffffc90044797738 R12: ffffffffc07b9bd0
> Jul 20 02:17:54 lamb kernel: [21061.422583] R13: 000000000000000d R14:
> 0000000000000000 R15: 000000000000000d
> Jul 20 02:17:54 lamb kernel: [21061.423052] FS: 0000000000000000(0000)
> GS:ffff888292840000(0000) knlGS:0000000000000000
> Jul 20 02:17:54 lamb kernel: [21061.423518] CS: e030 DS: 0000 ES: 0000 CR0:
> 0000000080050033
> Jul 20 02:17:54 lamb kernel: [21061.423986] CR2: 00007f909a037c30 CR3:
> 000000010d2dc000 CR4: 0000000000050660
> Jul 20 02:17:54 lamb kernel: [21061.424472] Call Trace:
> Jul 20 02:17:54 lamb kernel: [21061.424943] nvme_queue_rq+0x98/0x190 [nvme]
> Jul 20 02:17:54 lamb kernel: [21061.425425]
> blk_mq_dispatch_rq_list+0x123/0x7d0
> Jul 20 02:17:54 lamb kernel: [21061.425904] ? sbitmap_get+0x66/0x140
> Jul 20 02:17:54 lamb kernel: [21061.426385] ? elv_rb_del+0x1f/0x30
> Jul 20 02:17:54 lamb kernel: [21061.426909] ?
> deadline_remove_request+0x55/0xc0
> Jul 20 02:17:54 lamb kernel: [21061.427373]
> __blk_mq_do_dispatch_sched+0x164/0x2d0
> Jul 20 02:17:54 lamb kernel: [21061.427843]
> __blk_mq_sched_dispatch_requests+0x135/0x170
> Jul 20 02:17:54 lamb kernel: [21061.428310]
> blk_mq_sched_dispatch_requests+0x30/0x60
> Jul 20 02:17:54 lamb kernel: [21061.428795] __blk_mq_run_hw_queue+0x51/0xd0
> Jul 20 02:17:54 lamb kernel: [21061.429269]
> __blk_mq_delay_run_hw_queue+0x141/0x160
> Jul 20 02:17:54 lamb kernel: [21061.429752]
> blk_mq_sched_insert_requests+0x6a/0xf0
> Jul 20 02:17:54 lamb kernel: [21061.430233]
> blk_mq_flush_plug_list+0x119/0x1b0
> Jul 20 02:17:54 lamb kernel: [21061.430756] blk_flush_plug_list+0xd7/0x100
> Jul 20 02:17:54 lamb kernel: [21061.431241] blk_finish_plug+0x21/0x30
> Jul 20 02:17:54 lamb kernel: [21061.431734] dispatch_rw_block_io+0x6a5/0x9a0
> [xen_blkback]
> Jul 20 02:17:54 lamb kernel: [21061.432220] __do_block_io_op+0x31d/0x620
> [xen_blkback]
> Jul 20 02:17:54 lamb kernel: [21061.432714] ?
> _raw_spin_unlock_irqrestore+0x14/0x20
> Jul 20 02:17:54 lamb kernel: [21061.433193] ? try_to_del_timer_sync+0x4d/0x80
> Jul 20 02:17:54 lamb kernel: [21061.433680] xen_blkif_schedule+0xda/0x670
> [xen_blkback]
> Jul 20 02:17:54 lamb kernel: [21061.434160] ? __schedule+0x2c6/0x770
> Jul 20 02:17:54 lamb kernel: [21061.434679] ? finish_wait+0x80/0x80
> Jul 20 02:17:54 lamb kernel: [21061.435129] ? xen_blkif_be_int+0x30/0x30
> [xen_blkback]
> Jul 20 02:17:54 lamb kernel: [21061.435571] kthread+0x116/0x130
> Jul 20 02:17:54 lamb kernel: [21061.436002] ? kthread_park+0x80/0x80
> Jul 20 02:17:54 lamb kernel: [21061.436422] ret_from_fork+0x22/0x30
> Jul 20 02:17:54 lamb kernel: [21061.436846] ---[ end trace 1d90be7aea2d9148
> ]---
> Jul 20 02:17:54 lamb kernel: [21061.437250] blk_update_request: I/O error,
> dev nvme0n1, sector 912000815 op 0x1:(WRITE) flags 0x800 phys_seg 13 prio
> class 0
> Jul 20 02:17:54 lamb kernel: [21061.446344] md/raid1:md4: Disk failure on
> nvme0n1, disabling device.
> Jul 20 02:17:54 lamb kernel: [21061.446344] md/raid1:md4: Operation
> continuing on 1 devices.
I've tested and can confirm that this issue is fixed by:
https://lore.kernel.org/patchwork/patch/1442338/
which is in 5.14 already and was also backported to 5.10.50.
Debian buster-backports currently ships with 5.10.40; bullseye is
5.10.46; Salvatore Bonaccorso said on debian-kernel list that they
would backport this fix to all the 5.10 packages.
Thanks,
Andy
