** Changed in: linux (Ubuntu Focal)
Importance: Undecided => Medium
** Changed in: linux (Ubuntu Focal)
Status: In Progress => Fix Committed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2081085
Title:
wbt:wbt_* trace event NULL pointer dereference with GENHD_FL_HIDDEN
disks
Status in linux package in Ubuntu:
Invalid
Status in linux source package in Focal:
Fix Committed
Bug description:
[Impact]
Systems with storage devices that utilize the GENHD_FL_HIDDEN flag, such as
NVMe disks declaring support for multiple controllers (aka native
multipathing), will have a request queue with backing_dev_info->dev set to
NULL. When tracing is enabled with any of the wbt:wbt_* events enabled, a NULL
pointer dereference will occur in the corresponding trace function called from
wb_timer_fn. This occurs when the trace function attempts to access the
device's name with dev_name.
On a DGXA100 system, this can be reproduced by running the following, where
/dev/nvme0n1 is one of the 4 NVMe disks in the system that support native
multipathing:
$ echo 1 | sudo tee /sys/kernel/tracing/events/wbt/enable
$ echo 1 | sudo tee /sys/kernel/tracing/tracing_on
$ sudo dd if=/dev/zero of=/dev/nvme0n1
A NULL pointer dereference will occur and the system will become
unresponsive.
[Fix]
The upstream commit d51cfc53ade318 ("bdi: use bdi_dev_name() to get device
name") resolves this by changing the wbt:wbt_* trace functions to use the
bdi_dev_name function instead of dev_name. The bdi_dev_name function safely
handles the case where the supplied device is NULL.
[Test Case]
Verified that the commit d51cfc53ade318 ("bdi: use bdi_dev_name() to get
device name") resolves the issue on DGXA100 when applied to the
"Ubuntu-5.4.0-196.216" tag. The reproducer no longer causes a NULL pointer
dereference or otherwise crash the system.
[Regression Potential]
There is a low risk of a regression:
* In the focal K5.4 kernel, the bdi_dev_name function is used in other trace
event functions for the same purpose of catching the case where bdi->dev is
NULL.
* This change is already present in kernel versions 5.7 and newer.
[Other]
The patch d51cfc53ade318 ("bdi: use bdi_dev_name() to get device name") is
already present in Jammy K5.15 and newer.
-----
Originally found via kernel regression testing and reported here by
@cypressyew: https://bugs.launchpad.net/ubuntu-kernel-tests/+bug/2072972.
[ 1073.837085] BUG: kernel NULL pointer dereference, address: 0000000000000050
[ 1073.844858] #PF: supervisor read access in kernel mode
[ 1073.850589] #PF: error_code(0x0000) - not-present page
[ 1073.856318] PGD 0 P4D 0
[ 1073.859141] Oops: 0000 [#1] SMP NOPTI
[ 1073.863226] CPU: 9 PID: 0 Comm: swapper/9 Tainted: P OE
5.4.0-196-generic #216-Ubuntu
[ 1073.873319] Hardware name: NVIDIA DGXA100 920-23687-2530-000/DGXA100, BIOS
1.25 08/31/2023
[ 1073.882547] RIP: 0010:trace_event_raw_event_wbt_timer+0x6f/0x100
[ 1073.889248] Code: 59 80 e5 02 0f 85 8f 00 00 00 4c 89 e6 ba 34 00 00 00 48
8d 7d a0 e8 00 aa c9 ff 49 89 c4 48 85 c0 74 37 49 8b 87 b8 03 00 00 <48> 8b 70
50 48 85 f6 74 45 49 8d 7c 24 08 ba 20 00 00 00 e8 c9 18
[ 1073.910200] RSP: 0018:ffffaf08598a8da0 EFLAGS: 00010282
[ 1073.916029] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000080000100
[ 1073.923988] RDX: ffff970f0ba9501c RSI: 0000000000000100 RDI:
ffff970f0ba95018
[ 1073.931947] RBP: ffffaf08598a8e08 R08: ffff970f0ba95010 R09:
0000000000000100
[ 1073.939906] R10: ffffcf083fdc9a58 R11: 0000000000000386 R12:
ffff970f0ba9501c
[ 1073.947864] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff976efdfba000
[ 1073.955824] FS: 0000000000000000(0000) GS:ffff970f0f640000(0000)
knlGS:0000000000000000
[ 1073.964850] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1073.971259] CR2: 0000000000000050 CR3: 0000001f7de80000 CR4:
0000000000340ee0
[ 1073.979217] Call Trace:
[ 1073.981942] <IRQ>
[ 1073.984188] ? show_regs.cold+0x1a/0x1f
[ 1073.988456] ? __die+0x90/0xd9
[ 1073.991862] ? no_context.isra.0+0x12c/0x320
[ 1073.996626] ? update_group_capacity+0x2c/0x1d0
[ 1074.001679] ? __bad_area_nosemaphore+0x45/0x1a0
[ 1074.006829] ? bad_area_nosemaphore+0x16/0x20
[ 1074.011687] ? do_user_addr_fault+0x267/0x440
[ 1074.016547] ? __enqueue_entity+0x96/0xa0
[ 1074.021018] ? enqueue_entity+0x139/0x670
[ 1074.025490] ? __do_page_fault+0x58/0x90
[ 1074.029861] ? do_page_fault+0x2c/0xe0
[ 1074.034044] ? page_fault+0x34/0x40
[ 1074.037933] ? trace_event_raw_event_wbt_timer+0x6f/0x100
[ 1074.043954] ? enqueue_entity+0x139/0x670
[ 1074.048426] wb_timer_fn+0x1d6/0x3c0
[ 1074.052413] ? blk_mq_tag_update_depth+0x100/0x100
[ 1074.057755] blk_stat_timer_fn+0x13a/0x140
[ 1074.062326] call_timer_fn+0x32/0x130
[ 1074.066409] __run_timers.part.0+0x180/0x280
[ 1074.071174] ? tick_sched_handle+0x33/0x60
[ 1074.075740] ? tick_sched_timer+0x3d/0x80
[ 1074.080211] ? recalibrate_cpu_khz+0x10/0x10
[ 1074.084971] ? ktime_get+0x3e/0xa0
[ 1074.088765] ? native_apic_msr_write+0x2b/0x30
[ 1074.093719] run_timer_softirq+0x2a/0x50
[ 1074.098091] __do_softirq+0xd1/0x2c1
[ 1074.102078] irq_exit+0xae/0xb0
[ 1074.105580] smp_apic_timer_interrupt+0x7b/0x140
[ 1074.110728] apic_timer_interrupt+0xf/0x20
[ 1074.115295] </IRQ>
[ 1074.117634] RIP: 0010:native_safe_halt+0xe/0x10
[ 1074.122685] Code: 7b ff ff ff eb bd 90 90 90 90 90 90 e9 07 00 00 00 0f 00
2d e6 0c 50 00 f4 c3 66 90 e9 07 00 00 00 0f 00 2d d6 0c 50 00 fb f4 <c3> 90 0f
1f 44 00 00 55 48 89 e5 41 55 41 54 53 e8 dd 3e 61 ff 65
[ 1074.143638] RSP: 0018:ffffaf0840387e70 EFLAGS: 00000246 ORIG_RAX:
ffffffffffffff13
[ 1074.152083] RAX: ffffffff90d0ab70 RBX: 0000000000000009 RCX:
0000000000000001
[ 1074.160042] RDX: 0000000000065c72 RSI: 0000000000000083 RDI:
0000000000000009
[ 1074.168002] RBP: ffffaf0840387e90 R08: 0000000000000001 R09:
0000000000022f80
[ 1074.175962] R10: ffff980f3fed5328 R11: 0000000000000000 R12:
0000000000000009
[ 1074.183920] R13: ffff970f0bff5e00 R14: 0000000000000000 R15:
0000000000000000
[ 1074.191882] ? __cpuidle_text_start+0x8/0x8
[ 1074.196546] ? default_idle+0x20/0x140
[ 1074.200726] arch_cpu_idle+0x15/0x20
[ 1074.204711] default_idle_call+0x23/0x30
[ 1074.209085] do_idle+0x1fb/0x270
[ 1074.212683] ? complete+0x49/0x50
[ 1074.216376] cpu_startup_entry+0x20/0x30
[ 1074.220750] start_secondary+0x178/0x1d0
[ 1074.225126] secondary_startup_64+0xa4/0xb0
[ 1074.229789] Modules linked in: nls_iso8859_1 dm_multipath scsi_dh_rdac
scsi_dh_emc scsi_dh_alua amd64_edac_mod edac_mce_amd kvm_amd kvm nvidia_uvm(O)
nvidia_drm(PO) nvidia_modeset(PO) ipmi_ssif nvidia(PO) input_leds mlx5_ib(OE)
ib_uverbs(OE) binfmt_misc ib_core(OE) ccp k10temp ipmi_si ipmi_devintf
ipmi_msghandler mac_hid sch_fq_codel msr ramoops reed_solomon efi_pstore
ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov
async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0
multipath linear ses enclosure crct10dif_pclmul crc32_pclmul
ghash_clmulni_intel aesni_intel mlx5_core(OE) ast crypto_simd drm_vram_helper
cryptd glue_helper ttm drm_kms_helper pci_hyperv_intf syscopyarea mlxdevm(OE)
auxiliary(OE) sysfillrect mpt3sas hid_generic igb uas tls sysimgblt usbhid
raid_class mlxfw(OE) psample scsi_transport_sas usb_storage dca hid fb_sys_fops
i2c_algo_bit nvme mlx_compat(OE) drm nvme_core i2c_piix4
[ 1074.323033] CR2: 0000000000000050
[ 1074.326858] ---[ end trace f3c3dea0291e7e7a ]---
[ 1074.472012] RIP: 0010:trace_event_raw_event_wbt_timer+0x6f/0x100
[ 1074.478714] Code: 59 80 e5 02 0f 85 8f 00 00 00 4c 89 e6 ba 34 00 00 00 48
8d 7d a0 e8 00 aa c9 ff 49 89 c4 48 85 c0 74 37 49 8b 87 b8 03 00 00 <48> 8b 70
50 48 85 f6 74 45 49 8d 7c 24 08 ba 20 00 00 00 e8 c9 18
[ 1074.499668] RSP: 0018:ffffaf08598a8da0 EFLAGS: 00010282
[ 1074.505497] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
0000000080000100
[ 1074.513459] RDX: ffff970f0ba9501c RSI: 0000000000000100 RDI:
ffff970f0ba95018
[ 1074.521418] RBP: ffffaf08598a8e08 R08: ffff970f0ba95010 R09:
0000000000000100
[ 1074.529378] R10: ffffcf083fdc9a58 R11: 0000000000000386 R12:
ffff970f0ba9501c
[ 1074.537336] R13: 0000000000000000 R14: 0000000000000001 R15:
ffff976efdfba000
[ 1074.545297] FS: 0000000000000000(0000) GS:ffff970f0f640000(0000)
knlGS:0000000000000000
[ 1074.554325] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1074.560733] CR2: 0000000000000050 CR3: 0000001f7de80000 CR4:
0000000000340ee0
[ 1074.568695] Kernel panic - not syncing: Fatal exception in interrupt
[ 1074.578317] Kernel Offset: 0xf200000 from 0xffffffff81000000 (relocation
range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1075.126325] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2081085/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
